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I.  INTRODUCTION 


A.  INTRODUCTION 

The  United  States  Department  of  Defense  (DoD)  recently  reported  that  the 
development  costs  of  72  weapons  programs  had  climbed  40  percent  from  their 
initial  estimates,  there  was  an  average  delay  of  21  months,  and  the  total  systems 
overrun  was  $2  billion  dollars  (Government  Accountability  Office,  2008).  Studies 
show  that  these  development  problems  are  typically  not  caused  by  technology 
issues  but  are  largely  due  to  program  management  (Office  of  the  Under 
Secretary  of  Defense,  2000).  Improving  program  management  should  be  a 
primary  focus  of  the  DoD  if  there  is  to  be  any  hope  of  significantly  increasing 
program  performance.  One  of  the  key  aspects  of  the  DoD’s  program 
management  is  the  management  of  system  software  development. 

Software  has  become  such  an  integral  part  of  weapon  systems  that  it  is 
virtually  impossible  to  find  a  weapons  system  today  that  does  not  contain 
mission-critical  software  at  its  core  (Welby,  2010).  This  is  not  just  isolated  to  the 
DoD.  Reliance  on  software  keeps  growing  in  industries  as  diverse  as  transport, 
medical,  communications,  energy,  space,  entertainment,  and  finance  (Allen, 
2009).  As  the  world  increasingly  relies  on  software-intensive  systems,  there  will 
be  an  increased  need  for  effective  software  project  management  in  order  to  field 
successful  systems.  Ineffective  software  project  management  in  these  industries 
is  among  the  main  reasons  for  failures  in  software  projects  (Jones,  2004). 

Effective  Software  Project  management  is  crucial  to  a  software  project’s 

success.  It  was  observed  by  DeMarco  and  Lister  that  for  the  overwhelming 

majority  of  bankrupt  projects,  there  was  not  a  single  technological  issue  to 

explain  the  failure  (DeMarco  &  Lister,  1999).  Another  study  in  the  last  decade 

asserted  that  a  project  was  never  seen  to  fail  for  technical  reasons.  It  was  always 

human  failures  that  caused  otherwise  good  projects  to  grind  to  a  halt  (Robertson 

&  Robertson,  2005).  Despite  these  observations  most  software  engineering 
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research  emphasizes  technical  matters  above  behavioral  matters  (Glass,  2002). 
People  and  project  management  are  the  Achilles’  heel  of  software  projects. 

So  are  software  project  managers  just  poor  at  their  jobs  and  solely  to 
blame  for  project  failures?  This  surely  cannot  be  the  case  as  many  project 
managers  are  outstanding  professionals.  But  software  itself  is  incredibly  complex 
and  so  is  the  management  of  its  creation.  A  striking  proportion  of  project 
difficulties  stem  from  people  failing  to  implement  known  best  practices  (The 
Royal  Academy  of  Engineering  and  The  British  Computer  Society,  2004).  To 
become  effective  at  software  project  management  requires  the  project  team  to 
learn  certain  practices  until  they  become  habits.  Good  project  managers  will 
continually  seek  ways  to  improve  their  methods  and  learn  from  experience.  But 
changes  in  how  software  is  managed  do  not  come  quickly  or  easily.  Any  project 
management  improvement  process  needs  to  be  approached  deliberately  and 
purposefully.  Project  managers  need  tools  to  help  them  improve  their  software 
project  management.  A  tool  that  measures  and  monitors  the  effectiveness  of 
software  project  management  can  be  used  to  identify  strengths  and  weaknesses 
and  guide  improvement  of  the  software  project  management  practices  in  place 
on  the  project.  Improving  technical  processes  alone  cannot  ensure  a  successful 
project  outcome. 

B.  STATEMENT  OF  THE  PROBLEM 

Effective  project  management  involves  measurement.  Project  managers 
measure  schedule,  progress,  expenditure,  effort,  productively.  These 
measurements  are  made  to  take  the  pulse  of  the  project,  in  order  to  improve  the 
project’s  health,  if  need  be.  But  since  poor  software  project  management  can 
increase  software  costs  more  rapidly  than  any  other  factor,  as  Boehm  declared, 
should  not  the  project’s  management  itself  be  measured  and  monitored?  Garcia 
and  Suarez  stated  that  project  management  practices  are  considered  the 
cornerstone  of  the  software  lifecycle  (Garcia  &  Suarez,  2007).  If  the  project 
management  practices  can  be  improved,  then  a  project  should  increase  its 
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chance  of  success.  However,  committing  a  project  to  a  significant  improvement 
effort  requires  a  thorough  understanding  of  where  the  project  is  and,  perhaps 
more  importantly,  where  the  project  needs  to  grow  (Grant  &  Pennypacker,  2006). 
The  problem  with  current  project  management  appraisal  methods  is  that  they 
take  a  long  time  to  make  an  assessment,  they  do  not  focus  on  people  and  they 
are  targeted  at  the  organizational  level.  For  this  reason,  project  managers  are  not 
completely  equipped  with  the  right  software  project  management  effectiveness 
measurement  tools. 

1.  Effort  of  Analyzing  Project  Management  Practices 

Effective  software  project  managers  should  appreciate  a  candid  review  of 
how  a  project  is  being  conducted  or  was  conducted.  As  humans,  we  learn  from 
our  mistakes  and  conducting  a  post-mortem  analysis  of  a  project  is  considered  a 
best  practice  by  many  software  professionals.  This  is  one  method  for  a  project 
manager  to  analyze  the  effectiveness  of  the  software  project  management  on  the 
project.  However,  it  was  found  by  Chemuturi  and  Cagely  Jr.  that  the  project  post¬ 
mortem  evaluation  is  often  skipped  (Chemuturi  &  Cagley  Jr.,  2010).  The  reasons 
for  this  could  be  that  the  time  is  considered  better  spent  on  other  income¬ 
generating  activities.  A  software  project  management  effectiveness 
measurement  tool  could  assist  with  the  post-mortem  activity  and  even  reduce  the 
time  it  takes  to  conduct  the  activity. 

2.  Project  Manager  Performance 

In  general  terms,  there  are  three  types  of  categories  of  project  managers: 
those  that  know  the  best  practices  and  apply  them,  those  that  know  them  and  for 
whatever  reason  do  not  apply  them,  and  finally  those  that  do  not  know  them. 
Surprisingly,  there  is  an  absence  of  collective  professionalism  in  the  industry,  as 
well  as  inadequacies  in  the  education  and  training  of  staff  at  all  levels  (The  Royal 
Academy  of  Engineering  &  The  British  Computer  Society,  2004).  The  software 
project  managers’  network  asserts  that  a  big  problem  in  software  projects  is  an 
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ill-equipped  project  manager  (Phillips,  2000).  A  software  project  management 
effectiveness  measurement  tool  could  help  project  management  professionals 
identify  practices  at  which  their  project  is  poor  or  even  practices  they  do  not  use 
at  all.  Even  experienced  managers  could  benefit  from  this  type  of  tool.  Due  to  the 
pressure  and  fog  of  war  of  software  projects,  one  can  forget  to  apply  best 
management  practices. 

3.  Maturity  Models  Lack  a  People  Focus 

Currently,  there  are  a  number  of  Maturity  Models  in  widespread  use  that 
can  be  used  to  appraise  a  project’s  processes  and  guide  improvement  efforts. 
While  these  models  assist  with  improving  some  software  project  management 
processes,  they  ignore  the  people  side  of  software  development.  The  first 
maturity  model  that  comes  to  most  people’s  minds  in  the  software  development 
industry  is  the  Capability  Maturity  Model  Integration  (CMMI)  brand.  It  seeks  to 
make  proven  software  principles  part  of  the  organization’s  culture  and  is  often 
used  to  rate  organizations’  software  development  capabilities.  To  most  people, 
there  is  little  doubt  that  adopting  the  specific  practices  recommended  by  CMMI 
will  improve  an  organization’s  ability  to  manage  software  projects.  However, 
technical  processes  alone  cannot  ensure  a  successful  software  project  outcome. 
CMMI-DEV-vl  .2  contains  a  process  area  that  focuses  on  project  management, 
but  this  process  area  is  devoid  of  management  practice  related  to  people 
(Phillips,  2000).  CMMI-DEV-vl  .2  focuses  on  an  organization’s  technical 
processes  and  not  its  highly  unpredictable  and  behavioral  components — people. 
The  project  management  practices  in  CMMI-DEV-vl  .2  are  only  one  compartment 
of  the  greater  software  project  management  framework.  This  concept  is 
illustrated  in  Figure  1 .  Software  project  management  is  about  people  and  not  just 
technical  processes. 


4 


Figure  1 .  Software  Project  Management 


C.  BACKGROUND  AND  NEED 

There  is  evidence  to  suggest  that  deficient  project  management  practices 
may  be  one  of  the  principal  causes  of  software  project  problems.  As  such,  there 
has  been  a  widespread  investment  in  project  management  education  and  tools 
as  organizations  strive  to  become  good  a  delivering  projects  successfully  (Grant 
&  Pennypacker,  2006).  There  has  been  avid  interest  in  the  creation  of  models 
that  provide  a  collection  of  best  practices  that  managers  can  compare  to  their 
organizations’  practices  in  order  to  guide  improvement.  The  front-runners  in  this 
area  are  the  Project  Management  Maturity  Models  (PMMM),  but  there  is  also 
promising  research  in  more  lightweight  software  project  management 
measurement  tools. 

1.  Project  Management  Maturity  Models 

Maturity  Models  have  spread  quickly  across  the  globe  in  the  last  two 
decades.  From  the  foundation  established  by  CMMI,  new  models,  dubbed 
PMMMs,  have  immerged  to  focus  on  the  project  management  maturity  of 
organizations.  The  majority  of  PMMMs  work  in  a  similar  way  to  the  CMMI 
models.  PMMMs,  however,  are  concerned  with  generic  project  management  and 
do  not  focus  specifically  on  software  project  management.  Software  project 
management  is  more  different  from  traditional  project  management  than  most 
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professional  managers  expect  (Fairley,  2009).  There  are,  in  fact,  very  few 
Software  Project  Management  Maturity  Models  (SPMMM)  in  existence  today. 

While  an  SPMMM  provides  a  means  to  assess  the  level  of  the  software 
project  management  effectiveness,  it  does  have  a  few  limitations.  A  maturity  or 
capability  level  can  only  be  obtained  after  an  independent,  outside  group 
examines  the  organization  against  specific  criteria.  To  make  an  appraisal  of  an 
organization  usually  requires  preparation,  on-site  activities  and,  finally,  reporting. 
This  takes  a  considerable  amount  of  time.  Additionally,  maturity  models  claim  to 
be  able  to  target  specific  projects  but  are  really  focused  at  the  organizational 
level  and  although  maturity  models  have  exploded  onto  the  market  there  are 
many  organizations  that  are  still  not  using  them  (Garcia  &  Suarez,  2007).  Due  to 
these  limitations,  there  is  a  need  for  more  tailored,  lightweight  software  project 
management  effectiveness  measurement  tools.  A  lightweight  appraisal  tool  can 
be  used  in  a  lot  less  time  than  a  maturity  model  and  can  identify  ineffective 
project  management  practices  in  place  on  a  software  project. 

2.  Software  Project  Management  Effectiveness  Metric 

One  such  lightweight  measurement  tool  was  proposed  by  Demir  in  his 
dissertation  entitled  Measurement  of  Software  Project  Management 
Effectiveness  (Demir,  2008).  Demir  proposed  a  Software  Project  Management 
Evaluation  Model  (SPMEM)  that  provided  a  standard  quantitative  measure  of 
software  project  management  effectiveness.  The  model  accepted  input  data 
obtained  from  the  application  of  a  questionnaire  to  a  software  development 
project.  It  produced  a  standard  quantitative  measure,  between  zero  and  ten,  by 
comparing  the  practices  in  place  on  the  project  to  the  best  practices  in  the  model. 
Demir  measured  sixteen  software  projects  and  produced  a  software  project 
management  effectiveness  metric  score  for  each.  Pearson  product  moment 
correlation  analysis  was  performed  for  the  metric  scores  and  a  subjective  rating 
of  the  projects’  success.  It  was  found  that  there  was  a  strong  positive  correlation 
with  the  project  success  rating  and  the  software  project  management 
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effectiveness  metric  score.  In  addition,  half  of  the  variation  in  project  success 
could  be  explained  by  the  metric.  Both  of  these  findings  indicate  that  the  metric 
has  a  strong  practical  and  theoretical  foundation  to  build  upon. 

The  measurement  takes  significantly  less  time  to  perform  than  a  maturity 
model  appraisal  and  can  be  used  to  assist  in  the  postmortem  activity  of  a 
software  project.  The  measurement  can  identify  weak  project  management 
practices  on  a  project  and  can  guide  future  improvement  efforts.  It  can  guide 
managers  by  providing  a  quick  assessment  of  how  the  project  stands  against 
software  project  management  best  practices  contained  in  the  model.  When  the 
tool  is  used  to  measure  and  monitor  a  project,  it  can  act  as  a  reminder  not  to  let 
certain  practices  fall  by  the  wayside.  It  can  also  provide  objective  proof  of  the 
project’s  deficiencies  so  as  to  prove  to  stakeholders  what  improvement  efforts 
must  be  made  and  should  be  resourced. 

The  Software  Project  Management  Effectiveness  Metric,  while  promising, 
is  still  in  a  developmental  stage.  The  sample  size  of  16  projects  used  in  Demir’s 
study  is  not  statistically  significant.  In  addition,  the  previous  sample  included  very 
few  failed  projects.  Conducting  further  measurements  using  the  tool  will  provide 
more  insight  into  the  applicability  and  limitations  of  the  metric. 

D.  PURPOSE  OF  THE  STUDY 

1.  Purpose  Statement 

The  purpose  of  this  study  was  to  measure  the  software  project 
management  effectiveness  of  software  projects  using  the  SPMEM  in  order  to 
increase  the  pre-existing  sample  size  and  reassess  the  correlation  between  the 
software  project  management  effectiveness  metric  score  and  the  subjective 
project  success  rating. 

The  hypothesis  to  be  tested  is: 

The  success  of  a  software  project  positively  correlates  to  its 

software  project  management  effectiveness  metric  score. 
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If  having  a  high  software  project  management  effectiveness  metric  score 
is  associated  with  a  high  project  success  rating,  it  would  indicate  that  improving  a 
project’s  score  would  improve  the  project’s  chance  of  success. 

2.  Need/Rationale  for  the  Study 

The  software  project  management  effectiveness  metric  has  the  potential 
to  assist  project  managers  who  are  put  in  charge  of  software  intensive  system 
developments.  The  metric  can  assist  with  the  post-mortem  analysis  of  software 
projects,  via  identifying  areas  for  improvements  on  subsequent  projects.  The 
metric  can  provide  quantitative  evidence  to  support  improvement  process 
decisions  rather  than  just  going  off  of  a  project  managers  gut  feel.  This  tool  can 
be  used  to  measure  and  monitor  projects  so  that  project  managers  do  not  let 
best  practices  fall  out  of  favor  on  the  project. 

3.  Description  of  the  Study 

In  order  to  provide  an  assessment  of  the  correlation  between  the  project 
success  rating  and  the  metric,  data  from  recent  software  development  projects 
was  collected.  The  data  was  collected  using  the  Software  Project  Management 
Evaluation  Instrument  (SPMEI).  The  SPMEI,  which  is  a  comprehensive 
questionnaire,  was  administered  to  software  project  managers,  technical 
managers,  software  developers  and  team  leaders.  The  research  subjects  also 
provided  a  subjective  project  success  rating.  The  data  collected  using  the  SPMEI 
was  used  as  input  to  the  Software  Project  Management  Evaluation  Model 
(SPMEM).  This  model  used  the  raw  data  from  the  subjects’  responses  and 
produced  the  software  project  management  effectiveness  (PME)  score  for  each 
project.  These  two  measures  were  used  to  test  the  research  hypothesis.  In  order 
to  understand  the  measure  of  association  between  these  two  metrics,  a 
parametric  correlation  analysis  was  conducted.  The  testing  of  the  hypothesis  was 
conducted  by  analyzing  the  Pearson  product  moment  correlation  coefficient 
(PMCC)  between  the  two  measures. 
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4.  Expected  Goals  and  Outcomes  of  the  Study 


The  goal  of  this  study  was  to  build  upon  the  sample  of  sixteen  software 
projects  in  Demir’s  SPMEI  research.  With  a  larger  sample  size,  a  stronger 
argument  can  be  made  to  use  or  not  use  the  metric.  Another  goal  was  to  gain 
further  insight  into  the  usefulness  and  applicability  the  metric. 

E.  RESEARCH  QUESTIONS 

It  was  stated  by  Jones  that  effective  project  management  is  a  determinant 
in  the  success  of  the  software  projects  (Jones,  2004).  The  purpose  of  the  metric 
is  to  monitor  and  improve  the  effectiveness  of  software  project  management.  The 
following  questions  will  be  addressed  in  this  study. 

1)  Will  improving  a  project’s  PME  score  increase  the  project’s  chance 

of  success? 

(a)  What  is  the  relationship  between  the  PME  score  (measured) 
and  the  project  success  rating  (measured)? 

(b)  What  is  the  relationship  between  the  PME  score  (measured) 
and  the  size  of  the  project  (measured)? 

(c)  What  is  the  relationship  between  an  institution’s  CMMI  level 
(measured  variable)  and  the  PME  metric  (measured 
variable)? 

2)  What  are  software  development  practitioner’s  perceptions  towards 

the  practicality  and  usefulness  of  the  metric? 

(a)  What  are  software  development  practitioner’s  perceptions 
towards  the  manageability,  meaningfulness,  actionability, 
ambiguity,  reliability,  accuracy,  timeliness  and  predictability 
of  the  metric? 

(b)  Will  software  development  practitioners  use  the  metric? 
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F.  SIGNIFICANCE  TO  THE  FIELD 

From  the  literature  review  conducted  in  this  study,  it  became  evident  that 
the  software  engineering  field  contains  only  limited  scientific  work  that  addresses 
theories  of  measuring  software  project  management  effectiveness.  The  results  of 
this  study  have  helped  substantiate  the  applicability  and  usefulness  of  the  SPME 
metric.  The  projects  surveyed  in  this  study  also  benefited  by  receiving  metric 
scores  that  identified  areas  of  weakness  in  their  software  project  management. 

G.  DEFINITIONS 

Project  Success  Rating:  A  subjective  ranking,  on  a  scale  of  zero  to  ten, 
made  by  a  member  of  the  project  team  on  the  successfulness  of  a  project  (zero 
being  a  complete  failure  and  10  being  a  complete  success). 

Effectiveness:  Efficiency  is  doing  things  right.  Effectiveness  is  doing  the 
right  things. 

Conceptual  Framework:  A  set  of  theories  widely  accepted  enough  to 
serve  as  the  guiding  principles  within  a  particular  discipline. 

Project:  A  group  of  coordinated  work  activities  and  tasks  that  utilizes 
resources  to  achieve  specified  objectives  within  a  prescribed  time  frame  (Fairley, 
2009). 

Software  Project:  A  project  concerned  with  developing  software  for  a 
software  intensive  system.  Software  intensive  systems  include  one  or  more 
digital  devices  and  associated  software. 

Software  Project  Management:  The  collection  of  work  activities 
concerned  with  planning  and  estimating,  measuring  and  controlling,  coordinating 
and  leading,  and  managing  risk  factors  for  a  software  project  (Fairley,  2009). 
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Best  Practices:  Best  practices  are  reusable  activities  or  processes  that 
continuously  add  value  to  the  deliverables  of  the  project.  Best  practices  can  also 
increase  the  likelihood  of  success  of  each  and  every  project.  But  while  all  that 
sounds  good,  there  exists  a  fundamental  question  of  who  defines  what  is  or  is 
not  a  best  practice  (Kerzner,  2004). 

Process:  The  steps  taken  to  develop  software;  a  recipe  for  software.  A 
way  of  accomplishing  one  or  more  work  activities  and  tasks;  typically  involves 
procedures  and  the  use  of  software  tools  (Fairley,  2009). 

Product:  The  product  is  the  project’s  final  outcome.  Products  include 
software,  documentation,  and  training  and  maintenance  services  (Phillips,  2000). 

H.  LIMITATIONS 

Further  development  and  modification  to  improve  the  SPMEI  and  SPMEM 
were  considered  outside  of  the  scope  of  this  research.  The  SPMEI  was  applied 
to  only  nine  projects  due  to  the  difficulty  of  finding  suitable  participants  willing  to 
participate.  The  study  was  conducted  on  a  sample  of  convenience.  Having  a 
small  sample  size  reduces  the  studies’  external  validity  because  of  the  limited 
generalizability  to  other  settings  and  groups. 

I.  ETHICAL  CONSIDERATIONS 

As  this  study  involved  human  subjects,  the  research  required  approval 
from  the  Naval  Postgraduate  School’s  Institutional  Review  Board  (IRB)  to  ensure 
that  the  research  was  conducted  in  an  ethical  manner.  Due  to  the  nature  of  the 
research,  the  risk  to  participants  was  considered  low.  A  breach  of  a  subject’s 
confidentiality  may  have  resulted  in  some  embarrassment.  Informed  consent  was 
obtained  from  all  participants  and  the  consent  form  is  contained  in  Appendix  B. 
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II.  LITERATURE  REVIEW 


Many  research  initiatives  have  emerged  that  focus  on  the  improvement  of 
software  development  processes  and  the  technology  used  during  software 
development.  However,  one  area  often  underestimated  but  crucial  for  every 
software  development  effort  is  project  management.  (Mandl-Striegnitz  &  Litcher, 
1998).  Software  developers  cannot  rely  solely  on  technological  advances  to 
achieve  better  outcomes  in  the  development  of  software  products.  Software 
development  houses  need  to  make  significant  advances  in  the  way  they  conduct 
project  management  in  order  to  achieve  better  results.  Applicable  and  viable 
theories  on  software  project  management  need  to  be  discussed  and  developed, 
and  models  and  tools  need  to  be  tested  and  put  into  practice.  Only  then  can 
software  projects  achieve  better  outcomes.  One  of  the  most  important  steps,  for 
personnel  practicing  project  management,  is  to  look  in  the  mirror  and  identify  how 
their  software  project  management  practices  can  be  improved.  This  research  has 
identified  several  tools  available  in  open  literature  that  assess  and  measure  the 
effectiveness,  quality  and  maturity  of  software  project  management  practices. 
Before  these  tools  are  discussed,  a  brief  theory  of  software  project  management 
measurement  is  presented. 

A.  METRICS  IN  SOFTWARE  PROJECT 

Metrics  serve  only  one  purpose.  We  measure  to  manage  (Brotby,  2009). 
In  the  management  of  software  projects,  it  is  widely  accepted  as  best  practice  for 
managers  to  measure  different  components  of  their  projects.  For  instance, 
progress  is  measured  using  Earned  Value  Management  (EVM),  while  a  product’s 
performance  is  measured  by  using  Key  Performance  Indicators  (KPI)  and 
software  metrics.  Quantitative  measurements  are  essential  in  software 
engineering  and  there  is  a  constant  effort  from  academia  and  industry  to  improve 
and  discover  useful  metrics. 
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A  software  metric  is  a  measure  of  some  property  of  a  piece  of  software  or 
its  specifications  (Singh,  2009).  To  give  an  example,  here  are  some  software 
metrics  in  widespread  use: 

•  Number  of  Source  lines  of  code 

•  Faults  per  lines  of  code 

•  Number  of  lines  of  customer  requirements 

•  Function  Points 

•  Cyclomatic  complexity 

•  Program  load  time 

The  goal  of  research  concerning  software  metrics  is  to  obtain  objective, 
reproducible  and  quantifiable  measurements  of  software  products.  Metrics, 
measures,  and  monitoring  processes  exist  only  to  provide  decision  support 
(Brotby,  2009).  These  measurements  can  then  be  used  on  software  projects  to 
assist  with  schedule  and  budget  planning,  cost  estimation,  and  software 
performance  optimization.  The  measurements  can  also  be  used  to  predict  trouble 
ahead,  such  as  the  popular  faults  per  lines  of  code  metric. 

However,  simply  measuring  the  technical  aspects  of  the  software  itself  is 
only  one  part  of  a  much  larger  and  complex  project.  Effective  project 
management  is  also  a  determinant  in  the  success  of  the  software  projects 
(Jones,  2004).  Measuring  and  monitoring  the  behavioral  and  management  side 
of  a  software  project  should  also  be  able  to  assist  in  providing  decision  support.  If 
a  project  can  measure  and  monitor  its  software  project  management  capabilities, 
then  the  project  can  take  active  steps  to  improve  these  critical  practices. 
Measurement  of  one’s  software  project  management  effectiveness  enables  the 
improvement  of  practices  that  are  known  to  lead  to  a  greater  chance  of  project 
success. 

B.  SOFTWARE  PROJECT  MANAGEMENT  EFFECTIVENESS 

It  can  be  strongly  argued  that  the  effectiveness  of  software  project 
management  contributes  significantly  to  the  outcome  of  a  software  project.  But 
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just  what  is  software  project  management  effectiveness?  Effectiveness  is  defined 
in  the  Merriam-Webster  dictionary  as  the  power  to  produce  a  desired  effect 
(Merriam-Webster  Inc,  2011).  Based  on  this,  the  following  definition  is  offered: 
Software  Project  Management  Effectiveness  is  the  power  of  the  software  project 
management  practices  in  place  to  accomplish  the  objectives  of  the  software 
project. 

In  management,  effectiveness  relates  to  getting  the  right  things  done 
(Drucker,  1993).  If  the  right  software  project  management  practices  are  in  place 
and  the  practices  are  implemented  well,  then  the  software  project  management  is 
effective.  An  alternate  definition  is:  Software  Project  Management  Effectiveness 
is  the  degree  to  which  the  right  project  management  practices  are  in  place  to 
produce  the  intended  or  expected  result  of  the  software  project. 

C.  THEORY  OF  SOFTWARE  PROJECT  MANAGEMENT  EFFECTIVENESS 

MEASUREMENT 

The  theory  presented  in  this  thesis  proposes  that  it  is  possible  to  measure 
software  project  management  effectiveness  by  determining: 

•  if  the  right  software  project  management  practices  were  in  place 
during  a  project 

•  how  well  the  software  project  management  practices  were 
implemented 

The  right  software  project  management  practices  are  reusable  activities  or 
processes  that  continuously  add  value  to  the  deliverables  of  the  software  project 
(Kerzner,  2004).  By  implementing  these  practices,  a  software  project  can 
increase  the  likelihood  of  success. 

A  generic  conceptual  approach  for  measuring  software  project 
management  effectiveness  in  this  way  is  presented  in  Figure  2.  This  approach 
requires  the  development  of  a  software  project  management  framework  that 
describes  best  practices.  A  data  collection  instrument  must  then  be  developed  to 
comprehensively  sample  a  project  relative  to  the  previously  developed 
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framework.  Data  is  collected  using  the  instrument  and  analyzed  in  a  systematic 
way  by  the  software  project  evaluation  model  to  determine  the  score  of  the 
project’s  management  effectiveness.  The  project  can  then  take  action  to  improve 
areas  in  which  it  is  deficient. 
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Figure  2.  Conceptual  Approach  to  Software  Project  Effectiveness 

Measurement 


1.  Development  of  a  Software  Project  Management  Framework 

For  the  measurement  of  software  project  management  effectiveness  to 
work,  there  needs  to  be  a  perfect  standard  of  software  project  management  to 
measure  against.  While  all  that  sounds  good,  there  exists  a  fundamental 
question  of  who  defines  what  is  or  is  not  a  project  management  best  practice 
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(Kerzner,  2004).  There  is  no  one  correct  way  to  manage  a  project,  due  to  the 
extremely  complex  and  diverse  nature  of  software  projects.  What  works  well  as 
best  practice  for  one  may  not  work  equally  well  for  another.  For  the  measurement 
to  have  practicality,  there  needs  to  be  a  framework  of  effective  software  project 
management  practices  that,  if  implemented  by  a  project,  will  increase  the  chance 
of  success.  There  are  various  bodies  of  knowledge  on  the  theory  and  practice  of 
software  project  management  that  can  be  used  to  develop  such  a  framework. 
The  development  of  a  framework  for  software  project  management  is  the  first 
step  in  creating  an  objective  and  repeatable  metric. 

2.  Development  an  Instrument  and  Evaluation  Model 

Secondly,  a  data  collection  instrument(s)  must  be  developed  to  sample 
the  software  project  management  practices  of  a  project  in  a  representative  and 
comprehensive  manner.  In  this  study,  instrument  validity  is  the  extent  to  which 
the  data  collection  instrument  samples  the  effectiveness  of  the  software  project 
management  in  a  representative  and  comprehensive  manner.  Data  must  be 
collected  so  that  it  can  be  analyzed  to  identify  if  the  project  is  performing 
practices  as  suggested  by  the  project  management  framework. 

There  are  a  variety  of  data  collection  instruments  and  methods  that  can  be 
used  for  examining  a  software  project.  These  include  questionnaires,  interviews, 
and  documentation  reviews,  to  name  a  few.  Each  has  its  own  advantages  and 
disadvantages.  Questionnaires  were  the  most  commonly  used  instruments 
observed  in  this  literature  review.  This  is  mainly  because  questionnaires  can  be 
applied  to  many  people  and  projects  in  a  cost  effective  manner.  Questionnaires 
are  not  as  invasive  as  an  interview  and  can  provide  quantitative  data  that  can  be 
analyzed  promptly. 

3.  Collecting  and  Analyzing  Software  Project  Management  Data 

After  project  data  is  collected  by  the  instrument(s),  it  is  analyzed  to 
discover  how  well  the  software  project  management  practices  in  place  correlate 
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to  the  practices  defined  by  the  framework.  The  software  project  management 
evaluation  model  presented  in  Figure  3  is  used  to  systematically  analyze  the  data 
and  produce  a  quantitative  metric.  The  metric  will  give  an  indication  on  the 
effectiveness  of  the  project  management  practices  in  place.  Once  it  has  been 
determined  where  the  project  is  in  reality  compared  to  the  suggested  framework, 
a  report  can  be  generated  to  explain  to  the  project  where  their  project 
management  deficiencies  are. 


Figure  3.  Conceptual  Black  Box  Diagram  of  Software  Project  Management 

Evaluation 


4.  Making  Improvements 

The  project  reviews  the  report  and  the  metric  produced  in  order  to  develop 
their  own  action  plan  that  aims  to  implement  new  management  practices  or 
improve  existing  ones.  By  improving  their  practices  they  should  improve  their 
metric  score.  There  are  two  ways  that  the  metric  could  be  used;  and  these  are 
shown  graphically  in  Figure  4.  For  a  project  that  has  a  long  duration,  multiple 
measurements  of  the  software  project  management  effectiveness  can  be  made 
at  periodic  intervals  to  ensure  that  improvements  are  being  made.  For  a  project 
of  shorter  duration,  one  measurement  can  be  made  at  the  end  of  the  project  as 
part  of  a  post-mortem  process  so  that  improvements  can  be  made  by  the  project 
management  and  implemented  on  their  next  project. 
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Figure  4.  Measurement  Timings 

The  theory  of  the  measurement  of  software  project  management 
effectiveness  is  summarized  in  Table  1.  A  literature  review  was  conducted  on 
open  sources  to  identify  studies  that  are  related  to  the  concept  of  measuring  the 
effectiveness  of  software  project  management  as  presented  in  this  section.  The 
literature  review  identified  that  very  few  studies  have  been  published  that 
concern  this  topic.  Six  such  studies  are  summarized  and  discussed  in  the 
following  section. 
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Table  1.  Software  Project  Management  Effectiveness  Measurement 


Software  Project  Management  Effectiveness  Measurement 


What  is  being 
measured? 

>  The  software  project  management  effectiveness: 

■  The  amount  of  “right”  management  practices  that 
are  in  place? 

■  How  well  the  management  practices  in  place  are 
being  implemented? 

Why  is  it  measured? 

>  The  chance  of  software  project  success  is  greater 
when  effective  project  management  practices  are 
done. 

What  does  it  mean? 

>  A  high  score  means  that  the  project  has  implemented 
the  right  practices  and  is  performing  them  to  a  high 
degree  in  relation  to  the  theoretical  software  project 
management  framework.  The  project  has  a  higher 
chance  of  success  than  a  project  with  a  lower  score. 

>  A  low  score  means  that  the  project  has  not 
implemented  the  right  management  practices  in 
accordance  with  the  theoretical  software  project 
management  framework.  The  project  has  a  lower 
chance  of  success  than  a  project  with  a  high  score. 

Who  are  the 
Recipients? 

>  Software  project  management  practitioners 

What  action  is 
required? 

>  The  project  team  must  implement  changes  to  their 
project  management  practices  in  the  areas  where  they 
are  deficient  in  order  to  improve  their  chances  of  future 
project  success 

D.  RELATED  WORK 

1.  Study:  Software  Project  Management  Maturity  Assessment 
Model  (2007) 

In  their  paper  “Software  Project  Management  Maturity  Assessment  Model 
to  assess  the  level  of  Software  Project  Management  Practices,”  Fuazi  and  Ramli 
presented  a  model  to  assess  software  project  management  practices  using  their 
Software  Project  Management  Maturity  Assessment  (SPMMA)  model  (Ramli, 
2007).  The  purpose  of  the  SPMMA  was  to  help  a  company  measure  the  strength 
and  weaknesses  of  its  software  project  management  and  develop  action  plans  to 
make  improvements. 
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a.  Framework 

The  SPMMA  was  developed  using  the  concepts  defined  by  the 
Capability  Maturity  Model  Integration  (CMMI)  and  Software  Process 
Improvement  and  Capability  Determination  (SPICE)  assessment  models.  The 
framework  only  focuses  on  the  project  planning,  project  monitoring  and  control 
and  risk  management  process  areas.  The  research  was  considered  to  be  a  pilot 
program  and  these  three  process  areas  are  not  deemed  to  be  a  completely 
comprehensive  software  project  management  framework. 

b.  Data  Collection  Instruments 

There  were  two  types  of  data  collection  instruments  used  in  the 
model,  a  questionnaire  and  a  set  of  interview  questions.  The  questionnaire  was 
used  to  gather  data  indirectly  from  practitioners.  The  questions  were  organized 
into  groups  of  process  areas  drawn  from  the  previously  mentioned  framework, 
such  as  project  planning  and  risk  assessment.  The  respondents  could  select 
from  four  possible  answers  for  each  question:  yes,  no,  does  not  apply  and  don’t 
know.  An  extract  of  the  questions  for  the  risk  management  section  are  presented 
in  Table  2. 

Besides  the  questionnaires,  interviews  were  used  to  directly  obtain 
data  on  the  software  project  management  practices.  The  interview  was  used  to 
give  the  assessor  a  better  understanding  of  the  project  management  practices. 
Related  project  management  documentation  was  also  reviewed  to  gain  a  more 
thorough  understanding  of  the  project.  An  extract  of  the  interview  questions  is 
presented  in  Table  3. 
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Table  2.  Questionnaire:  Risk  Management  Section 


Questionnaire:  Risk  Management  Section 

Are  risks  contingency  activities  planned? 

Yes 

No 

Does 

Not 

Apply 

Don’t 

Know 

Does  the  project  conduct  meetings  to  identify 
common  causes  of  defects? 

Yes 

No 

Does 

Not 

Apply 

Don’t 

Know 

Once  identified,  are  common  causes  of  risks 
prioritized  and  systematically  eliminated? 

Yes 

No 

Does 

Not 

Apply 

Don’t 

Know 

Does  the  project  follow  a  written 
organizational  policy  for  risks  management 
activities? 

Yes 

No 

Does 

Not 

Apply 

Don’t 

Know 

Do  members  of  the  software  engineering 
group  and  other  software-related  groups 
receive  required  training  to  perform  their  risks 
prevention  activities? 

Yes 

No 

Does 

Not 

Apply 

Don’t 

Know 

Table  3.  Interview  Extract 


Interview  Extract 


Please  tell  me  about  yourself  and  your  experience  as  it  relates  to  this  project? 
Please  describe  your  role  and  responsibilities  on  the  project? 

How  do  you  know  what  you  are  supposed  to  be  working  on? 

What  training  have  you  had  for  your  job? 

Are  you  involved  with  any  of  the  estimating  and  planning  of  the  software  project? 


c.  Data  Collection  and  Analysis 

The  SPMMA  was  carried  out  on  one  mid-size  Information 
Technology  (IT)  Company.  Based  on  the  questionnaire  responses,  interviews 
and  discussions  among  the  assessment  team,  a  rating  of  fully  implemented, 
largely  implemented,  partially  implemented  or  not  implemented  was  provided  for 
each  of  the  three  process  areas.  Additionally,  the  assessment  team  produced  a 
final  report  on  the  assessment  findings  and  made  improvement 
recommendations. 
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d.  Results  and  Summary 


The  pilot  program  received  the  following  ratings  for  the  three  project 
management  areas: 

•  Project  planning  -  largely  implemented 

•  Project  monitoring  and  control  -  largely  implemented 

•  Risk  management  -  partially  implemented 

A  recommendation  was  made  to  the  project  to  establish  proper  risk 
identification  and  contingency  list.  Fuazi  and  Ramli  concluded  that  the  SPMMA 
could  be  used  as  a  tool  to  measure  the  level  of  maturity  of  the  software  project 
management  practices  in  an  organization  (Ramli,  2007).  While  the  SPMMA 
presents  a  method  to  measure  the  strength  and  weaknesses  of  an  organization’s 
software  project  management  there  are  a  few  concerns.  First,  only  one  project 
consisting  of  40  personnel  was  assessed.  Additionally,  the  tool  only  assesses 
project  management  in  three  areas  and  gives  each  area  one  of  four  possible 
ratings.  The  three  areas  in  this  framework  are  not  considered  comprehensive 
and  the  ratings  do  not  provide  much  granularity.  A  summary  of  the  model  is 
provided  in  Table  4. 
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Table  4.  Summary  of  SPMMA 


Summary  of  SPMMA 


What  is  being 
measured? 

>  The  maturity  of  the  organizations  software  project 
management  practices 

Why  is  it  measured? 

>  To  help  the  organization  measure  the  strength  and 
weaknesses  of  its  software  project  management 
and  develop  action  plans  to  make  improvements 

What  does  it  mean? 

>  Fully  implemented  -  affirmation  exists  to  confirm 
the  implementation  of  the  project  management 
practice  area  and  no  weaknesses  are  noted 

>  Largely  implemented  -  affirmation  exists  to  confirm 
the  implementation  of  the  project  management 
practice  area  and  one  or  more  weaknesses  are 
noted 

>  Partially  implemented  -  affirmations  suggest  that 
some  aspects  of  the  project  management  practice 
are  implemented  and  one  or  more  weaknesses  are 
noted 

>  Not  implemented  -  no  other  evidence  supports  the 
conclusion  that  the  project  practice  is  implemented 

Who  are  the 
Recipients? 

>  Project  management 

What  action  is  required?  >  The  assessed  project  has  to  prepare  an  action  plan 

that  specifies  how,  when  and  by  whom  each 
recommendation  is  to  be  implemented 


2.  Study:  What  Project  Management  Practices  Lead  to  Success 
(2005) 

Although  not  a  measurement  model,  Verner  and  Evanco  conducted 
relevant  research  using  a  questionnaire,  in  an  attempt  to  determine  the  factors 
that  lead  to  successful  projects.  They  claimed  that  quantitative  survey  based 
research  regarding  software  development’s  early,  non-technical  aspects  is 
lacking  (Verner  &  Evanco,  2005). 


24 


a.  Framework 

To  develop  their  software  project  management  framework,  Verner 
and  Evanco  conducted  wide  ranging,  structured  discussions  with  21  senior 
software  practitioners  to  document  views  regarding  the  software  project 
management  practices  they  considered  important. 

b.  Data  Collection  Instruments 

A  questionnaire  was  developed  on  the  basis  of  these  discussions. 
The  questionnaire  was  organized  into  seven  project  management  areas 
composed  of  numerous  questions.  An  extract  of  the  questionnaire  is  provided  in 
Table  5.  Respondents  were  also  asked  if  they  considered  the  project  successful. 


Table  5.  Verner  and  Evanco’s  Questionnaire 


Verner  and  Evanco’s  Questionnaire 

Did  the  project  have  a  project  manager? 

Yes 

No 

Was  the  PM  above  average? 

Yes 

No 

Was  the  PM  experienced  in  the  applications  area? 

Yes 

No 

Did  the  PM  understand  the  customer’s  problems? 

Yes 

No 

Did  the  PM  communicate  well  with  the  staff? 

Yes 

No 

Were  requirements  gathered  using  a  specific  method? 

Yes 

No 

Were  requirements  complete  and  accurate  at  the  project’s 
start? 

Yes 

No 

c.  Data  Collection  and  Analysis 

In  total,  122  in-house  software  development  projects  were  analyzed 
using  the  questionnaire.  The  sample  was  not  random,  but  rather  a  convenience 
sample  of  practitioners  that  Verner  and  Evanco  knew.  The  sample  size  was  very 
large  for  software  engineering  research  of  this  nature  and  was  the  largest  sample 
size  discovered  in  this  literature  review.  The  variables  in  the  survey  were 
analyzed  for  correlation  with  project  success  and  failure. 
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d.  Results  and  Summary 


The  objectives  of  Verner  and  Evanco’s  research  differ  from  those  of 
the  research  in  this  thesis.  Instead  of  attempting  to  measure  the  effectiveness  of 
the  software  project  management  practices  in  place  on  a  project  the  empirical 
research  attempted  to  identify  project  management  failures  that  lead  to  success 
or  failure.  It  was  found  that  a  clear  vision  of  the  final  product,  good  requirements, 
active  risk  management  and  post-mortem  reviews  can  all  help  increase  the  odds 
of  success  (Verner  &  Evanco,  2005).  For  all  projects,  changing  the  project 
manager  was  significantly  negatively  correlated  with  project  success.  If 
requirements  were  initially  incomplete,  completing  them  during  the  project  was 
positively  associated  with  success.  Because  software  developers  were  surveyed, 
the  results  were  limited  to  their  knowledge,  attitudes  and  beliefs  regarding  the 
projects  and  Project  Managers  with  which  they  were  involved.  The  method 
followed  in  Verner  and  Evanco’s  research  is  an  excellent  way  of  developing  a 
solid  software  project  management  framework. 

3.  Project  Management  Maturity:  An  Assessment  of  Project 
Management  Capabilities  Among  and  Between  Selected 
Industries  (2006) 

Committing  an  organization  to  a  significant  improvement  effort  requires  a 
thorough  understanding  of  where  the  organization  is  and,  perhaps  more 
importantly,  where  the  organization  needs  to  grow  (Grant  &  Pennypacker,  2006). 
One  way  to  address  this  need  is  via  the  use  of  project  management  maturity 
models.  The  emergence  of  the  project  management  maturity  model  can 
generally  be  traced  to  the  Capability  Maturity  Model  developed  by  the  Software 
Engineering  Institute  (SEI)  at  Carnegie  Mellon  (Skulmoski,  2001).  Project 
management  consulting  firms  have  played  a  leadership  role  in  the  development 
of  many  models,  largely  because  the  models  are  designed  to  identify  areas  upon 
which  improvement  efforts  should  focus.  There  are  currently  over  30  models  in 
existence  (Grant  &  Pennypacker,  2006). 
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A  typical  model  works  by  assessing  an  organization's  project  management 
maturity.  Once  the  initial  level  of  maturity  and  areas  for  improvement  are 
identified,  the  model  provides  a  roadmap,  outlining  the  necessary  steps  to  take 
toward  project  management  maturity  advancement.  Grant  and  Pennypacker 
conducted  research  to  determine  the  level  of  project  management  maturity  based 
on  42  detailed  components  among  a  wide  range  of  industries. 

a.  Framework 

The  research  conducted  used  the  Project  Management  Solutions 
Incorporated’s  Project  Management  Maturity  Model  (PMMMsm).  The  model 
adopts  a  two-dimensional  framework,  as  shown  in  Figure  5.  The  first  dimension 
reflects  the  level  of  maturity  and  is  based  on  the  structure  of  the  SEI  capability 
maturity  model.  The  second  dimension  depicts  the  key  areas  of  project 
management  addressed.  This  dimension  adopts  the  structure  of  the  PMI’s  nine 
knowledge  areas  (Project  Management  Institute,  2000).  Each  of  the  nine 
knowledge  areas  were  further  broken  down  into  key  components  that  provide  for 
a  more  rigorous  and  specific  determination  of  the  project  management  maturity. 
There  were  42  components  in  total. 
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Each  Knowledge  Area  is  broken  down  into  specific 
components.  Specific  Components  are  used  to  measure 
maturity  and  develop  action  plans.  The  number  of 
components  associated  with  each  knowledge  area  is 
presented  parenthetically  following  the  title  of  the 
knowledge  area. 


Figure  5.  Project  Management  Maturity  Model  (From:  Grant  &  Pennypacker, 

2006) 


b.  Data  Collection  Instruments 

A  survey  was  generated  that  included  a  specific  question  for  each 
of  the  42  components  of  project  management  maturity.  To  ensure  the  content 
validity  of  the  survey  instrument,  the  CBP  Knowledge  Board  reviewed  it  during 
the  survey  development  process  (Grant  &  Pennypacker,  2006).  An  excerpt  of  the 
survey  is  provided  in  Table  6.  One  advantage  of  this  behaviorally  anchored 
response  scale  format  is  that  it  has  been  shown  to  reduce  leniency  bias,  or  the 
tendency  of  a  respondent  to  be  overly  generous  or  severe  in  evaluating 
organizational  performance  (Grant  &  Pennypacker,  2006). 
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Table  6.  Cost  Management:  Resource  Planning 


Cost  Management:  Resource  Planning 


Level  1 )  Project  managers  have  developed  their  own  way  of  identifying  resources 
and  quantities  needed;  functional  support  areas  are  sometimes  overlooked; 
process  is  not  documented  and  varies  by  project. 

Level  2)  Complete  resource  listing  is  for  all  labor  categories,  equipment,  and 
material;  planning  process  is  developed  and  documented  to  include  the  resource 
listing  and  methodologies  for  determining  quantities;  planning  process  is 
supported  by  management  and  is  becoming  accepted  throughout  the 
organization. 

Level  3)  Planning  process  is  fully  implemented  within  the  organization;  project’s 
resource  requirements  are  uploaded  into  the  project  office’s  resource  repository. 
Level  4)  All  processes  are  in  place,  documented,  and  being  fully  utilized;  process 
is  fully  integrated  with  the  project  office  and  the  human  resources  project 
management  process. 

Level  5)  An  improvement  process  is  in  place  to  continuously  improve  resource 
planning  to  completely  identify  all  requirements  as  early  as  possible  in  the  right 
quantities;  lessons  learned  are  captured  and  used  to  improve  resource  planning 
efforts. 


c.  Data  Collection  and  Analysis 

A  total  of  126  organizations  were  surveyed  using  a  web-based 
survey.  Each  of  the  126  respondents  was  asked  to  rate  the  project  management 
maturity  of  his  or  her  organization  with  respect  to  42  specific  components  of 
project  management  maturity.  Nearly  67%  of  respondents  indicated  their 
organizations  were  operating  at  level  1 — initial  processes  (13.7%)  or  at  level  2 — 
structured  process  and  standards  (53.2%).  While  a  notable  portion  of 
respondents  rated  their  organizations  as  having  reached  level  3 — organizational 
standards  and  institutionalized  process  (19.4%),  a  mere  7.3%  indicated  their 
organizations  were  operating  at  level  4 — managed  process  and  only  6.5% 
assessed  their  organizations  as  having  achieved  level  5 — optimizing  process 
(Grant  &  Pennypacker,  2006). 
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d.  Results  and  Summary 


While  this  study  does  not  rigorously  measure  project  management 
maturity  in  the  participating  organizations,  it  did  serve  its  purpose  of  collecting  the 
ratings  of  numerous  organizations’  project  management  maturity.  However,  this 
research  focuses  on  organizational  project  management  maturity  and 
effectiveness,  which  is  related  remotely  to  project  management  effectiveness.  It 
is  also  concerned  with  the  higher  level  concept  of  project  management  and  not 
software  project  management. 

Maturity  in  project  management  is  the  development  of  systems  and 
processes  that  are  repetitive  in  nature  and  provide  a  high  probability  that  each 
project  will  be  a  success  (Kerzner,  2004).  It  was  found  that  there  were  many 
Project  Management  Maturity  Models  available  to  organizations  wishing  to 
improve  their  project  management.  These  models  focus  on  generic  project 
management  and  do  not  specifically  address  the  unique  attributes  of  software 
project  management. 
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Table  7.  Summary  of  PMMM 


Summary  of  PMMM 


What  is  being 
measured? 

>  Where  the  organization  is  concerning  their  project 
management  maturity 

Why  is  it  measured? 

>  A  thorough  understanding  of  where  the 
organization  is  and,  perhaps  more  importantly, 
where  the  organization  needs  to  grow  is  essential 
in  order  to  make  improvements 

What  does  it  mean? 

>  Level  1  -  Initial  (chaotic,  ad  hoc,  individual  heroics) 

-  the  starting  point  for  use  of  a  new  process 

>  Level  2  -  Managed  -  the  process  is  managed  in 
accordance  with  agreed  metrics 

>  Level  3  -  Defined  -  the  process  is 
defined/confirmed  as  a  standard  business  process, 
and  broken  down  to  levels  0,  1  and  2 

>  Level  4  -  Quantitatively  managed 

>  Level  5  -  Optimizing  -  process  management 
includes  deliberate  process 
optimization/improvement 

Who  are  the 

Recipients? 

>  Project  Managers  and  Executive  Management 

What  action  is  required? 

>  Organization  takes  steps  toward  project 
management  maturity  advancement  and 
performance  improvement 

4.  Study:  Quality  Management  Metric  (1999) 

Osmundson  et  al.  (2003)  developed  a  method,  called  the  Quality 
Management  Metric  (QMM),  to  measure  the  quality  of  software  management. 
The  QMM  is  a  composite  score  obtained  using  a  questionnaire  administered  to 
both  the  program  manager  and  a  sample  of  his  or  her  peers.  The  QMM  is 
intended  to  both  characterize  the  quality  of  software  management  and  be  used  to 
improve  an  individual’s  and  an  organization’s  software  project  management 
capabilities. 
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a.  Framework 

It  was  proposed  that  the  following  four  areas  collectively  were  a 
suitable  framework  for  the  basis  of  a  measurement  of  the  quality  of  the  software 
management  in  a  project: 

•  Requirements  management 

•  Estimation  &  planning  management 

•  People  management 

•  Risk  Management 

These  areas  were  validated  informally  by  experienced  software 
professionals  through  the  focus  groups  and  one-on-one  interviews. 

b.  Data  Collection  Instrument 

The  QMM  was  built  to  be  an  objective,  repeatable  metric  to 
determine  the  quality  of  the  software  management,  measure  improvement,  and 
predict  future  success  levels  of  projects.  A  two-part  questionnaire  was  developed 
to  quantitatively  measure  the  state  of  the  software  management  quality. 


Table  8.  Education/Planning  Management 


Estimation/Planning  Management:  choose  the  most  applicable  term  of  the 
two  for  each  row 

At  least  one  estimation  method  used  in 
program 

No  estimates 

Formal  derivation  of  product  metric  for 
estimation  of  size 

Ad  hoc  size  estimation 

Ad  hoc  process  evaluation 

Formal  derivation  of  at  least  one 
process  metric 

Develop  work  breakdown  structure 

Assign  work  as  needs  arise 

Estimates  are  developed  to  fulfill  a  data 
call  only 

Use  estimates  to  plan  program 

Use  estimates  to  sell  program  only 

Estimates  are  useful  to  the  project 
team  for  planning  purposes 

Expert  judgment  for  estimation 

Ad  hoc  estimates 
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The  questions  were  designed  to  confine  responses.  Part  one  of  the 
questionnaire  contained  pair  choice  questions  where  the  respondent  had  to 
choose  one  of  two  statements  that  best  describe  the  project.  An  extract  from  part 
one  of  the  survey  is  provided  in  Table  8.  Often,  the  pair  choices  were  repeated 
with  different  wording  to  confirm  earlier  choices  and  measure  the  strength  of  any 
tendencies.  Part  two  of  the  questionnaire  asks  for  one  of  three  responses:  yes, 
no  or  not  applicable.  This  format  standardized  the  response  for  easier 
comparison.  An  extract  of  part  two  of  the  survey  is  provided  in  Table  9. 

Table  9.  People  Management  Questionnaire 


People  Management  Questionnaire 

PM  is  accessible  in  person  by  each  team  member 

Yes 

No 

PM  is  accessible  via  email  by  each  team  member 

Yes 

No 

PM  is  accessible  via  phone  by  each  team  member 

Yes 

No 

PM  acts  as  facilitator  to  solving  personnel  conflicts 

Yes 

No 

PM  attempts  to  spotlight  individuals  in  the  program  for 
positive  exposure 

Yes 

No 

PM  maintains  regular  communication  with  users 

Yes 

No 

PM  must  approve  all  interactions  with  users 

Yes 

No 

c.  Data  Collection  and  Analysis 

The  survey  was  administered  to  13  projects  in  the  United  States 
Department  of  Defense  Environment.  The  projects  ranged  in  size  from  three 
software  developers  to  twenty-five  software  developers.  The  time  frame  of  the 
programs  surveyed  range  from  1992  to  2000. 

Each  choice  in  the  questionnaire  had  a  point  value  assigned  to  it 
based  on  the  relative  importance  of  the  question.  Point  totals  for  part  one  and 
part  two  were  then  added  together  to  determine  the  total  points  for  each  area  of 
software  project  management.  The  total  points  of  each  section  were  multiplied  by 
its  relative  importance  coefficient  to  yield  a  weighted  score.  After  weighted  scores 
were  determined  for  each  of  the  four  sections,  they  were  summed  together  to 
yield  the  QMM  score. 

<?««  -  0,92 RqM  +  Q,G7£Y PM  +  0,55 fffcM  + 1,6 GPM 
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d.  Results  and  Summary 


Each  respondent  was  also  asked  to  rate  the  success  of  their  project 
on  a  scale  of  zero  to  ten.  The  calculated  metric  from  each  of  the  projects  was 
compared  to  the  subjective  project  success  rating.  This  yielded  a  positive 
correlation  with  the  subjective  assessments  of  the  project  success. 

The  QMM  was  the  earliest  research  identified  by  this  literature 
review  to  deal  with  the  measurement  of  software  project  management 
effectiveness.  The  research  showed  promise  but  was  limited  by  the  sample  set 
only  consisting  of  Department  of  Defense  projects.  Additionally  the  projects  were 
all  during  the  1990s,  and  the  metric  has  been  further  validated  since  then. 


Table  10.  Summary  of  QMM 


Summary  of  QMM 

What  is  being 
measured? 

> 

Quality  of  software  management 

Why  is  it  measured? 

> 

Improve  organization’s  estimation  process  by 
including  management  quality  as  a  program 
attribute 

> 

Provide  feedback  to  software  program  managers 
as  to  their  management  effectiveness 

What  does  it  mean? 

> 

Highest  possible  score  -  100%  -  High  chance  of 

> 

program  success 

Lowest  possible  score  -  0%  -  Low  chance  of 

program  success 

Who  are  the 

Recipients? 

> 

Project  Manager 

What  action  is  required? 

> 

Improve  software  management  area  with  the 
lowest  score 

5.  Study:  Two  Phase  Questionnaire  (2007) 

Another  questionnaire  based-model  was  developed  by  Garcia  and  Suarez 
in  2007.  Their  approach  sought  to  obtain  a  baseline  snapshot  of  project 
management  practices  in  small-to-medium  enterprises  using  a  two-phase 

questionnaire  to  identify  both  performed  and  non-performed  practices  (Garcia  & 
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Suarez,  2007).  The  goal  was  to  identify  those  practices  that  are  performed  but 
not  documented,  that  practices  need  more  attention,  and  which  are  not 
implemented  due  to  bad  management  or  unawareness. 

a.  Framework 

To  obtain  an  accurate  picture  of  the  project  management  practices, 
Garcia  and  Suarez  based  their  framework  on  the  Capability  Maturity  Model 
Integration  for  Development  (CMMI-DEV)  (Software  Engineering  Institute,  2006). 
The  following  seven  well-established  project  management  areas  were  used  in 
the  construction  of  the  framework: 

•  Project  Planning 

•  Project  Monitoring  and  Control 

•  Requirements  Management 

•  Configuration  Management 

•  Process  and  Product  Quality  Assurance 

•  Supplier  Agreement  Management 

•  Measurement  and  Analysis 

b.  Data  Collection  Instruments 

A  questionnaire  was  developed  using  closed  questions  as  the  main 
instrument  for  collecting  data  on  the  proposed  framework.  It  was  argued  that  the 
application  of  a  questionnaire  to  an  organization’s  project  team  can  provide 
useful  information  related  to  the  current  state  of  the  project  management 
practices  and  indicate  those  that  required  immediate  attention.  The  questionnaire 
was  divided  into  two  phases.  This  division  is  mainly  due  to  the  fact  that  the 
CMMI-DEV  clearly  differentiates  between  specific  practices  and  generic 
practices.  Another  reason  for  the  division  into  two  phases  is  because  each 
section  is  applied  to  a  different  domain  of  people.  The  specific  practices  phase 
refers  to  the  series  of  steps  that  have  to  be  followed  to  perform  the  project 
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management  practices.  The  generic  practices  phase  refers  to  the  maturity  and 
institutionalization  of  the  project  management  practices  (Garcia  &  Suarez,  2007). 


c.  Data  Collection  and  Analysis 

The  respondent  could  choose  from  the  range  of  possible  answers 
provided  in  Figure  6.  Giving  a  specific  weight  to  each  response  was  proposed  to 
enable  the  easy  analysis  of  the  results  of  the  evaluation  and  identify  which 
practices  were  common  within  the  whole  organization  and  which  ones  were  not 
performed  at  all.  At  the  time  of  publication,  no  such  evaluation  was  undertaken. 


Possible 

Answer 

Peiform 

Level 

Desciiption 

The  activih’  is  documented  and  established  in  the 

Always 

4 

organization.  It  is  always  realized,  between  75  and 
100%  of  the  time,  in  organization  software  projects 

The  activity  is  established  in  the  organization  but 

Usually 

3 

rarefy  documented.  It  is  usually  realized,  between  50 
and  75  %  of  the  time,  in  organization  software 

projects 

Sometimes 

2 

The  activity’  is  weakly  established  in  the  organization. 

It  is  realized  sometimes,  between  25  and  50%  of  the 

time,  in  organization  software  projects 

Rarely  if 
ever 

1 

The  activity  is  rarely  performed  in  the  organization.  It 
is  rarefy  realized,  between  1  and  25  %  of  the  time,  in 
organization  software  projects 

Never 

0 

The  activity  is  not  performed  in  the  organization.  No 
person  or  group  performs  the  activity  in  the 

organization. 

Don’t 

Know 

The  person  is  not  sure  how  to  answer  the  question. 

Not  Apply 

The  question  is  nor  applicable  to  the  organization. 

Comments 

Tltis  space  is  for  elaborating  or  qualifying  one 's 
response  to  a  question,  and  it  is  mandatory’  when  one 
selects  Don  7  know  or  Not  Apply’  options. 

Figure  6.  Possible  Responses  in  Two  Phase  Questionnaire  (From:  Garcia  & 

Suarez,  2007) 


d.  Results  and  Summary 

Garcia  and  Suarez  felt  that  a  more  accurate  picture  of  the  project 
management  practices  of  an  organization  could  be  obtained  by  administering  a 
questionnaire.  The  next  step  in  their  research  was  related  to  the  validation  of  the 
questionnaire.  It  was  declared  that  in  the  future  their  questionnaire  would  be 
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administered  to  26  small-to-medium  enterprises  through  a  project  funded  by  the 
Spanish  Ministry  of  Industry,  Tourism,  and  Trade. 


6.  Software  Project  Management  Effectiveness  Metric  (2008) 

The  latest  research  known  to  cover  the  measurement  of  software  project 
management  effectiveness  was  published  by  Demir  (2008).  The  metric 
developed  by  Demir  sought  to  provide  a  standard  quantitative  measure  of 
software  project  management  effectiveness  from  the  start  of  a  project  to  its 
delivery.  The  objective  of  the  metric  was  to  help  managers  in  software 
development  organizations  to  evaluate,  monitor  and  improve  their  project 
management  effectiveness. 

a.  Framework 

A  software  project  management  framework  was  developed  by 
Demir,  and  was  validated  by  surveying  16  software  projects.  The  framework 
consisted  of  15  areas,  which  included:  communication,  teamwork,  leadership, 
organizational  commitment,  project  manager,  stakeholder  involvement,  staffing 
and  hiring,  requirements  management,  project  planning  and  estimation,  project 
monitoring  and  control,  scope  management,  configuration  management,  quality 
engineering,  risk  assessment,  and  risk  control. 

b.  Data  Collection  Instruments 

The  Software  Project  Management  Evaluation  Instrument  (SPMEI), 
which  was  a  comprehensive  questionnaire,  was  used  to  gather  project  data.  The 
data  collection  tool  was  used  to  gather  project  data  related  to  fifteen  project 
management  areas  of  the  framework. 

c.  Data  Collection  and  Analysis 

Twenty  software  projects  were  assessed  using  the  SPMEI  in  order 
to  investigate  the  applicability  and  limitations  of  the  metric.  A  member  of  the 
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project  organization  who  had  a  broad  knowledge  on  all  aspects  of  the  project 
management  was  asked  to  complete  the  questionnaire.  Then  the  data  gathered 
by  the  instrument  was  fed  into  the  software  project  management  evaluation 
model  (SPMEM).  Reponses  to  the  questions  were  assigned  specific  scores  in  a 
similar  way  to  the  QMM  mentioned  previously.  SPMEM  simply  combines  these 
scores  in  a  systematic  way  to  produce  a  score  for  each  project  management 
area  and  these  scores  are  then  used  to  compute  a  software  project  management 
effectiveness  (PME)  score  based  on  a  scale  from  0  to  10.  A  score  of  0  indicates 
the  least  effective  project  management,  while  a  score  of  10  indicates  the  most 
effective  project  management.  Each  respondent  was  also  asked  to  provide  a 
subjective  success  rating  from  0  to  10  in  the  same  way  as  the  QMM. 

d.  Results 

The  research  provided  empirical  evidence  required  for  the 
validation  of  the  metric.  A  Pearson  product  moment  correlation  analysis  on  the 
data  gathered  showed  that  there  is  a  strong  positive  correlation  with  success 
ratings  and  the  software  project  management  effectiveness  metric.  The  result  of 
the  analysis  on  the  data  indicated  that  half  of  the  variation  in  software  project 
success  may  be  explained  by  the  project  management  effectiveness  metric. 
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Table  11.  Summary  of  SPMEM 


Summary  of  SPMEM 

What  is  being 
measured? 

> 

Software  project  management  effectiveness 

Why  is  it  measured? 

> 

Software  project  success  is  dependent  on  effective 
software  project  management 

What  does  it  mean? 

> 

Highest  possible  score  -  10  -  A  high  PME  score 
indicates  a  high  probability  of  project  success 

> 

Lowest  possible  score  -  0  -  a  low  PME  score 
indicates  a  low  probability  of  project  success. 

Who  are  the 

Recipients? 

> 

Software  project  managers 

What  action  is  required? 

> 

Management  takes  steps  to  improve  their  software 
project  management  practices 

E.  SUMMARY  OF  MODELS 

From  the  six  studies  reviewed,  it  was  revealed  that  there  is  limited 
research  on  the  topic  of  the  measurement  of  software  project  management 
effectiveness.  All  of  the  studies  reviewed  are  summarized  in  Table  12.  Out  of  the 
six  studies,  only  three  provided  an  actual  methodology  to  measure  software 
project  management  effectiveness  or  maturity.  These  three  models  were  all  in 
early  developmental  stages. 

1.  Framework 

Each  study  established  a  framework  for  software  project  management, 
even  if  it  was  not  specifically  called  a  framework  in  the  study.  Three  of  the 
frameworks  were  based  upon  the  Software  Engineering  Institute’s  Capability 
Maturity  Models.  The  others  were  based  upon  research  and  validated  through 
peer  reviews. 

The  different  software  project  management  frameworks  varied  in  content 
and  comprehensiveness.  There  were,  however,  some  recurring  themes. 
Requirements  management  was  considered  important  in  four  of  the  frameworks; 
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planning  and  estimation  in  five;  risk  management  was  present  in  four  and 
monitoring  and  control  stood  out  in  three.  The  framework  for  the  SPMEM  was 
found  to  be  the  most  comprehensive  framework. 

2.  Data  Collection  Instruments 

A  constant  across  all  the  studies  was  the  use  of  a  questionnaire  to  gather 
data  on  the  project  management  practices.  In  each  study,  it  was  argued  that  the 
application  of  questionnaires  consumed  less  time,  effort  and  financial  resources 
than  other  methods  of  data  collection  such  as  interviews  and  document  reviews. 
Another  common  theme  was  that  the  questionnaires  were  written  in  such  a  way 
as  to  minimize  open-ended,  subjective  essay  type  answers.  Of  all  the  studies 
reviewed,  only  one  used  interviews  and  document  reviews  and  that  was  to 
complement  the  use  of  a  questionnaire  in  the  data  gathering  process. 

3.  Measurement 

The  SPMMA  only  provides  four  possible  ratings  for  the  maturity  of  the 
measured  project  management  areas.  The  QMM  provides  much  more 
granularity,  with  the  highest  possible  score  being  100%.  The  SPMEM  also 
offered  a  high  level  of  granularity,  with  an  ordinal  scale  of  0  to  100. 

4.  Time  to  Implement 

To  make  the  measurement  usable  by  practitioners  in  the  field,  data  needs 
to  be  gathered  quickly  and  easily.  The  QMM  was  the  quickest  metric  to 
implement  at  approximately  45  minutes,  followed  by  the  SPMEI  at  approximately 
90  minutes.  The  SPMMA  took  much  longer  to  get  a  result.  This  was  due  to  the 
interviews,  documentation  reviews  and  meetings  that  were  required  to  make  an 
assessment. 

5.  Sample  Size 

The  three  models  that  actually  involve  the  measurement  of  software 
project  management  maturity  are  in  their  early  stages  of  development.  The 
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SPMMA  was  tested  on  only  one  project.  The  QMM  was  used  on  13  United 
States  Department  of  Defense  projects.  The  SPMEI  was  applied  to  20  projects  of 
varying  sizes  and  industries. 

The  concept  of  the  PMMM  was  extended  by  the  SPMMA.  This  model 
focused  specifically  on  software  project  management.  However  where  these 
types  of  models  focus  on  assessing  the  organization,  the  theory  of  project 
management  effectiveness  is  concerned  with  measuring  the  software 
management  of  a  single  project  within  an  organization.  A  large  organization  may 
claim  to  have  a  project  management  maturity  level  of  4  when  they  have  multiple 
business  units  with  hundreds  of  projects.  Does  this  mean  that  every  business 
unit  and  every  project  operate  at  a  level  4?  This  is  possible  but  not  likely. 

At  the  completion  of  the  literature  review  the  measurement  methods  were 
subjectively  ranked  in  order  of  effectiveness  and  potential  for  future  use.  The 
results  are  shown  in  Table  12.  Out  of  the  studies  surveyed,  the  SPMEM  showed 
the  most  promise  for  the  measurement  of  software  project  management 
effectiveness.  The  framework  and  questionnaire  developed  were  the  most 
comprehensive  and  extensive.  The  measurements  made  thus  far  by  Demir  have 
shown  a  strong  positive  correlation  with  project  success.  The  time  to  implement 
the  questionnaire  is  reasonable  and  it  has  a  strong  sample  base  to  build  upon. 
The  SPMEM  is  reviewed  in  detail  in  the  following  chapter. 
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Table  12.  Summary  and  Ranking  of  Studies  in  the  Literature  Review 


Rank 

Model 

framework 

Instruments 

Ordinal  scale  of 
measurement 

Time 

(hrs) 

Sample 

size 

■  Communication 

■  Teamwork 

■  Leadership 

■  organizational  commitment 

■  project  manager 

■  stakeholder  involvement 

1 

SPMEM 

■  staffing  and  hiring 

■  requirements  management 

■  project  planning  and 
estimation 

Questionnaire 

116  questions 

0-100 

1.5 

16 

■  project  monitoring  and  control 

■  scope  management 

■  configuration  management 

■  quality  engineering 

■  risk  assessment 

■  risk  control 

2 

QMM 

■  Requirements  management 

■  Estimation  and  planning 

■  People  management 

■  Risk  Management 

Questionnaire 

0-100 

0.75 

13 

3 

SPMMA 

■  Project  Planning 

■  Project  Monitoring  and  Control 

■  Risk  Management 

Questionnaire 

Interview 

1-4 

-16.0 

1 

4 

Two  phase 

■  Project  Planning 

■  Project  Monitoring  and  Control 

■  Requirements  Management 

■  Configuration  Management 

■  Process  and  product  quality 
assurance 

■  Supplier  agreement 
management 

■  Measurement  and  analysis 

Questionnaire 

X 

-1.0 

0 

5 

PMMM 

■  Project  Integration 

Management 

■  Scope  Management 

■  Time  Management 

■  Cost  Management 

■  Quality  Management 

■  Project  Human  resource 
Management 

■  Communications 

Management 

■  Risk  Management 

■  Procurement  Management 

Questionnaire 

42  questions 

1-5 

-1.0 

126 
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6  In  house 


Project  Management 
Requirements  elicitation  and 
management 

Cost  and  effort  estimation  and 

scheduling 

Postmortem 


Questionnaire 


x  -0.25  122 
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III.  REVIEW  OF  THE  SOFTWARE  PROJECT  MANAGEMENT 

EFFECTIVENESS  METRIC 


Of  the  studies  reviewed  in  the  previous  chapter,  Demir’s  software  project 
management  effectiveness  metric  demonstrated  the  most  potential  as  a  software 
project  management  measurement  tool.  This  chapter  provides  a  more  detailed 
review  of  the  metric.  The  development  and  validation  of  the  software  project 
management  framework  used  for  the  metric  will  be  covered  and  the  data 
collection  instrument  and  the  software  project  management  evaluation  model  will 
also  be  discussed.  A  summary  of  the  results  obtained  by  Demir’s  research  will 
conclude  the  chapter. 

A.  3PR  SOFTWARE  PROJECT  MANAGEMENT  FRAMEWORK 

In  Demir’s  study,  a  simple  software  project  management  framework  was 
developed  that  collected  a  set  of  software  project  management  practices  to  serve 
as  guiding  principles  for  the  software  project  management  discipline.  The 
framework  was  developed  by  an  extensive  review  of  the  ubiquitous  project 
management  models,  bodies  of  knowledge,  standards  and  guidelines  in 
worldwide  circulation.  To  substantiate  the  developmental  framework  a  survey 
was  conducted  on  78  software  practitioners  from  around  the  world.  Demir’s 
framework  consists  of  four  main  software  project  management  areas:  people, 
process,  product  and  risk. 

•  People.  People  management  lies  at  the  core  of  software  project 
management  and  inclusion  in  the  framework  was  mandatory. 
Thomsett  (1995)  pointed  out  that  most  projects  fail  because  of 
people  and  project  management  concerns  rather  than  technical 
issues. 

•  Process.  The  CMMI  focus  is  on  improving  the  maturity  of 
organizations  by  improving  their  processes  (CMMI  Product  Team, 
2006).  The  process  main  area  focuses  on  key  software  project 
management  processes. 

•  Product.  The  software  product  is  considered  the  outcome  of  a 
software  project,  which  may  be  a  product,  service  or  result.  The 
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objective  of  a  project  is  to  create  a  product  with  which  the 
stakeholders  are  satisfied.  This  area  is  concerned  with  project 
management  practices  that  focus  their  attention  on  the  product 
quality. 

•  Risk:  Risk  management  is  an  inherent  aspect  of  any  software 
project.  Boehm  (1991)  indicated  that  in  most  software  project 
disasters,  the  problems  could  have  been  avoided  or  reduced  if  the 
high-risk  elements  had  been  identified  and  resolved  early  in  the 
process. 

The  framework  consists  of  areas  that  can  be  measured.  Each  main  area  is 
decomposed  into  sub  areas  of  project  management.  The  sub  areas  give  a  higher 
level  of  granularity  and  assist  in  more  refined  measurements.  Measurements  in 
the  sub  areas  can  help  project  managers  improve  specific  practices  that  are 
lacking.  The  complete  framework  is  displayed  in  Figure  7  and  is  called  the  3PR 
framework. 


Communication 

Teamwork 

Organizational 

Commitment 

Project  Manager 

Stakeholder 

Involvement 

Staffing  and 
Hiring 

Process 

Requirements  Management 

Project  Monitoring  and 
Control 

Project  Planning  and 
Estimation 

Scope  Management 

Product 

Quality  Engineering 

Configuration  Management 

Risk  Control 

Risk  Assessment 

Figure  7.  3PR  Software  Project  Management  Framework 


1.  People  -  Sub  Project  Management  Areas 

The  people  main  area  includes  seven  sub  areas  of  software  project 
management.  They  are  communication,  teamwork,  leadership,  organizational 
commitment,  project  manager,  stakeholder  involvement  and  staffing  and  hiring. 
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a.  Communication 

A  successful  project  requires  constant  and  effective  communication 
between  project  stakeholders.  It  is  a  prerequisite  to  getting  the  right  things  done 
in  the  right  way.  Sharing  knowledge  empowers  project  stakeholders.  Among  all 
the  project  management  areas  listed  in  the  Project  Management  Book  of 
Knowledge,  communications  has  the  largest  impact  on  project  results  (Muller, 
2003). 


b.  Teamwork 

Teamwork  is  the  process  through  which  a  collection  of  individuals 
cooperates  to  achieve  an  expressed  common  goal  (Rasing,  2011).  As  software  is 
developed  by  teams,  strong  teamwork  is  essential  to  successfully  completing  a 
software  project. 


c.  Leadership 

In  a  software  development  environment,  leadership  is  how 
personnel  in  management  positions  exert  social  influence  to  enlist  the  aid  and 
support  of  others  in  the  accomplishment  of  project  goals.  The  thing  great  leaders 
have  in  common  is  the  ability  to  get  the  right  things  done. 

d.  Organizational  Commitment 

In  the  framework  organizational  commitment  is  the  employee’s 
psychological  attachment  to  the  organization  and  organizational  goals  (Brown, 
2003). 


e.  Project  Manager 

The  project  manager  position  is  a  key  role  in  a  software  project’s 
organizational  structure.  A  project  manager  should  be  a  competent  manager  and 
leader. 
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f.  Stakeholder  Involvement 

The  stakeholder  engagement  sub  area  is  concerned  with  the  level 
of  involvement  of  all  the  different  stakeholders  during  the  project  development 
effort. 


g.  Staffing  and  Hiring 

In  this  framework,  staffing  and  hiring  is  the  ability  to  source  human 
resources  and  put  them  in  the  right  project  role.  Hiring  is  the  process  of 
employing  personnel  from  outside  the  organization,  whereas  staffing  is  the 
process  of  sourcing  personnel  from  within  the  organization. 

2.  Process — Sub  Project  Management  Areas 

This  sub  area  includes  requirements  management,  project  monitoring  and 
control,  project  planning  and  estimation,  and  scope  management.  These  areas 
are  more  closely  aligned  to  the  process  areas  in  the  CMMI-DEV  model  1 .3. 

a.  Requirements  Management 

This  process  involves  the  management  of  the  software 
requirements  and  is  not  to  be  confused  with  the  requirements  development 
process.  Requirements  must  be  controlled  and  consistency  of  requirements  must 
be  maintained  with  plans  and  work  products. 

b.  Project  Monitoring  and  Control 

Comparing  progress  to  plans  and  applying  corrective  action  as 
needed.  Project  monitoring  is  the  process  of  keeping  the  project,  project-related 
factors  and  project  metrics  under  continuous  observation.  Project  control  is  the 
process  of  ensuring  that  a  project  goes  according  to  what  was  planned. 
Deviations  from  the  plan  should  be  controlled  and  kept  to  a  minimum. 
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c.  Project  Planning  and  Estimation 

Project  planning  involves  establishing  and  maintaining  the  plans 
that  define  the  project  work  activities.  Software  project  estimation  includes 
establishing  estimates  of  project  cost,  schedule  and  resources  using  various 
methods,  techniques  and  tools. 

d.  Scope  Management 

This  is  the  process  of  defining  the  scope  of  the  project  and  keeping 
track  of  any  changes  to  the  scope.  Scope  management  was  found  in  the 
validation  of  the  framework,  explained  later,  to  be  the  most  challenging  sub  area 
of  the  software  project  management  framework. 

3.  Product — Sub  Project  Management  Areas 

This  main  area  includes  only  two  sub  areas:  configuration  management 
and  quality  engineering. 

a.  Configuration  Management 

Software  configuration  management  is  the  discipline  that  enables 
us  to  keep  evolving  software  products  under  control,  and  thus  contributes  to 
satisfying  quality  constraints  (Estublier,  2000).  Even  though  configuration 
management  is  a  process,  it  comes  under  this  main  area  because  it  focuses  on 
the  products  developed  by  a  software  project. 

b.  Quality  Engineering 

Quality  engineering  involves  all  activities  put  in  place  to  ensure  the 
development  of  a  high  quality  product.  In  this  framework,  quality  engineering  is 
not  quality  assurance.  Quality  engineering  includes  all  the  procedures  and 
processes  used  to  ensure  products  or  services  are  designed  and  produced  to 
meet  or  exceed  customer  requirements. 
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4.  Risk — Sub  Project  Management  Areas 

This  main  area  includes  only  two  sub  areas;  risk  assessment  and  risk 
control. 

a.  Risk  Assessment 

Identify  potential  problems.  According  to  Boehm  (1991),  risk 
assessment  involves  risk  identification,  risk  analysis  and  risk  prioritization. 

b  Risk  Control 

Develop  and  implement  strategies  and  techniques  for  mitigating 
them.  In  order  to  conduct  risk  control,  an  effective  risk  assessment  process  has 
to  be  in  place.  Risk  control  involves  risk  management  planning,  risk  resolution, 
and  risk  monitoring. 

Due  to  the  nature  of  project  management,  the  sub  areas  are  closely 
tied  to  each  other.  For  example,  an  effective  risk  control  can  only  be  achieved  as 
a  result  of  effective  risk  assessment.  Effective  teamwork  can  be  achieved  via 
effective  communication,  an  able  project  manager,  effective  leadership  of  various 
leaders  in  the  project  organization  and  commitment  from  stakeholders. 

5.  Validation  of  the  Software  Project  Management  Framework 

In  order  to  validate  the  framework,  a  survey  was  distributed  to  software 
development  practitioners  to  garner  opinions  on  the  framework.  This  form  of 
empirical  evidence  was  required  to  substantiate  the  framework. 

A  self-administered  questionnaire,  which  contained  thirteen  questions, 
was  developed  by  Demir.  The  purpose  of  the  questionnaire  was  to  identify  the 
importance  of  the  software  project  management  main  areas  and  sub  areas.  The 
survey  was  also  used  to  identify  challenging  areas  of  software  project 
management. 

The  survey  was  conducted  in  2007  and  was  delivered  to  approximately 

400  software  development  practitioners.  The  sample  was  random  and  80  usable 
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responses  were  obtained.  It  was  found  that  all  of  the  sub  areas  of  the  software 
project  management  framework  were  deemed  to  be  important  by  the  sample 
population.  On  a  seven-point  Likert  scale,  the  average  importance  ratings  ranged 
from  a  minimum  of  four  to  a  maximum  of  six.  This  indicated  that  all  of  the  areas 
were  felt  to  be  important  by  the  sample  population.  The  people  sub  areas  were 
rated  the  highest  and  the  process  sub  areas  were  the  second  highest.  The 
product  and  risk  sub  areas  were  rated  lower  than  the  others,  but  were 
indistinguishable  from  each  other. 

Additionally,  participants  were  asked  to  rate  the  importance  of  the  four 
main  areas  so  that  the  total  score  added  to  100.  The  mean  of  the  ratings  were 
the  following: 

•  People:  33.00% 

•  Process:  29.07% 

•  Product:  20.40% 

•  Risk:  17.53% 


These  ratings  were  used  to  adjust  the  software  project  management 
framework,  as  shown  in  Figure  8.  The  results  of  the  validation  study  guided  the 
development  of  the  software  project  management  evaluation  instrument  and 
evaluation  model. 
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Figure  8.  Adjusted  3PR  Framework 
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B.  SOFTWARE  PROJECT  MANAGEMENT  EVALUATION  INSTRUMENT 


The  goal  of  the  SPMEI  is  to  gather  data  on  what  happened  during  a 
software  project.  The  instrument  is  not  used  for  research  as  such  but  is  intended 
to  be  used  as  a  project  management  tool  on  software  projects.  The  SPMEI  data 
collection  instrument  is  a  self-administered  questionnaire,  consisting  of  the  fifteen 
sections  corresponding  to  the  fifteen  sub  project  management  areas  of  the  3PR 
framework.  Each  section  is  comprised  of  a  series  of  questions.  Each  question 
inquires  about  the  effectiveness  of  an  activity  or  an  entity  related  to  software 
project  management.  The  complete  instrument  contains  over  330  questions  and 
is  provided  in  Appendix  B. 

1.  Software  Project  Management  Evaluation  Instrument  Design 

In  every  project,  there  are  a  set  of  software  project  management  practices 

that: 

•  should  have  been  performed  and  were 

•  should  have  been  performed  and  were  not 

•  should  not  have  been  performed  but  were 

The  SPMEI  investigates  the  project  and  collects  data  on  all  three  of  these 
scenarios.  The  instrument  collects  data  on: 

•  The  existence  of  software  management  practices 

•  The  rigor  or  quality  of  the  practice 

Table  13  presents  the  different  sections  in  the  instrument  and  the  number 
of  questions  in  each  section.  In  a  questionnaire,  questions  can  be  classified  into 
open  and  closed  questions.  The  complexity  of  analyzing  data  provided  by  open 
questions  is  higher  than  those  in  closed  questions  (Yamanishi  &  Li,  2002). 
Closed  questions  provide  less  information  but  the  results  can  be  more  easily 
interpreted  in  a  measurement  model.  The  questions  in  SPMEI  are 
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closed  for  this  reason.  Closed  questions  also  reduce  the  time  required  to 
complete  the  survey.  No  one  wants  to  use  a  metric  that  takes  an  annoyingly  long 
time  to  produce. 

The  instrument  covers  the  activities  between  the  project  conception  and 
the  delivery.  Conception  is  the  point  where  the  project  was  established  and 
funded.  Delivery  is  the  point  in  time  where  the  final  product  is  delivered  to  the 
customer. 


Table  13.  SPMEI  Question  Break  Down 


Project  Management  Area 

Number  of  Questions 

Communication 

23 

Teamwork 

30 

Leadership 

17 

Organizational  Commitment 

26 

Project  Manager 

27 

Stakeholder  Involvement  (Market  or  Contract) 

12  or  16 

Staffing  and  Hiring 

29 

Requirements  Management 

27 

Project  Monitoring  and  Control 

19 

Project  Planning  and  Estimation 

35 

Scope  Management 

16 

Configuration  Management 

13 

Quality  Engineering 

20 

Risk  Control 

17 

Risk  Assessment  (With  Subcontracting  or  Without 
Subcontracting) 

20  or  19 

Total 

330-335 

2.  Application  of  the  Instrument 

a.  Who  Can  Use  the  Instrument? 

The  metric  is  likely  to  be  used  by  managers  and  organizations  that 

are  committed  to  achieving  better  results  from  their  projects.  These  types  of 

managers  and  organizations  value  candid  assessments  of  their  current  practices 

and  continuously  seek  to  make  improvements. 
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The  instrument  can  only  be  used  by  a  project  member  who  has 
extensive  knowledge  and  understanding  of  all  the  aspects  of  the  project 
management  practices.  Generally,  this  type  of  person  will  fill  the  following  roles  in 
a  software  development  project: 

•  Project  manager 

•  Team  leader 

•  Experienced  developer 

•  Software  architect 

b.  What  Projects  Can  Be  Measured  with  the  SPMEI? 

The  instrument  is  only  applicable  to  software  intensive  development 
projects.  The  instrument  is  not  restricted  to  either  public  or  private  sector 
projects.  The  instrument  is  not  applicable  to  corrective,  perfective  and  adaptive 
maintenance  efforts.  Managing  these  sets  of  activities  is  different  than  managing 
development  activities.  The  framework  and  instrument  were  not  designed  for 
software  maintenance  projects. 

c.  Temporal  Boundaries 

The  instrument  must  only  be  applied  to  projects  conducted  after 
1980  (Demir,  2008).  Until  the  nature  of  software  projects  changes  dramatically, 
the  instrument  may  continue  to  be  used  and  improved.  Demir  speculated  that  the 
metric  will  be  applicable  for  at  least  the  next  15  years.  The  use  of  the  metric  is 
applicable  to  projects  conducted  from  approximately  1980  to  2025. 

d.  When  Can  the  SPMEI  Be  Applied? 

The  project  must  be  established  for  a  certain  period  before  the 
instrument  can  be  applied.  The  earliest  that  the  measurement  can  be  made  is 
when  the  project  has  completed  the  initial  requirements  phase,  or  in  other  terms, 
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the  inception  or  conceptual  phase.  By  the  time  the  project  has  reached  this  point, 
many  of  the  essential  project  management  related  activities  are  already  in  place. 
To  be  specific,  the  following  must  be  in  place: 

•  the  project  manager  has  been  chosen 

•  the  project  organization  is  identified 

•  stakeholders  have  been  identified 

•  most  planning  and  estimation  activities  have  been  carried  out 

•  the  project  scope  has  been  established 

•  configuration  management  systems,  project  databases  and  other 
automated  systems  are  in  place 

•  quality  policy  is  in  place 

•  project  monitoring  and  control  procedures  should  be  in  place 

•  project  communication  procedures  should  be  in  place 

•  An  initial  risk  assessment  has  been  undertaken 


Table  14.  SPMEI  Summary 


Name  of  the  Instrument 

Software  project  management  evaluation 
instrument 

Acronym 

SPMEI 

Main  Use  of  Instrument 

Obtain  data  on  what  happened  during  the  project 
development 

Type  of  Instrument 

Self-administered  Questionnaire 

Participants 

>  Project  team  members  who  have  extensive 
knowledge  of  all  aspects  of  the  project. 

■  Executive  managers  overseeing  projects 

■  Project  managers 

■  Project  technical  managers 

■  Team  leaders 

Applicability 

>  Software-intensive  development  projects 

>  Applicable  to  any  project  organization  size 

>  Applicable  with  any  software  development  life- 
cycle  model 

>  Applicable  to  project  after  some  requirements 
development  activities  are  conducted 

Scope 

Project  start  to  project  delivery  (Project  start  is  the 
time  when  the  business  decision  is  made) 
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Number  of  Sections 

15 

Number  of  Questions 

330-335 

Type  of  Questions 

>  Multiple  choice 

>  Statements  with  a  psychometric  scale  (5-point 
Likert  item  based  on  agreement  to  a  statement) 

>  All  questions  are  closed  form 

Time  to  complete 

Average  of  90  minutes 

C.  SOFTWARE  PROJECT  MANAGEMENT  EVALUATION  MODEL 

The  SPMEM  and  the  SPMEI  were  developed  simultaneously.  The 
software  project  management  areas  in  the  previously  developed  framework 
correspond  to  the  variables  in  the  SPMEM  (Equation  1).  The  associated 
weighting  of  each  variable  was  determined  by  the  results  of  the  framework 
validation  survey.  The  variables  in  Equation  1  are  calculated  based  on  the  data 
gathered  from  the  SPMEI.  For  each  of  the  variables  (namely  the  software  project 
main  areas)  there  is  an  associated  model  to  determine  the  value.  Equations  2,  3, 
4  and  5  are  used  to  calculate  the  main  area  scores. 

1.  High-Level  Evaluation  Model 

The  high-level  evaluation  model  for  the  metric  is  as  follows: 

(l)  Mi1  Scvrv  -  0 .ZZPcovlsS  -I-  0.2507 ProceaaS  -I-  8.2O-y7’r0<£tcrt'5,-l-  0.1753i?fjA5 

where: 

PME  Score:  Software  Project  Management  Effectiveness  Score, 

PeopleS:  People  Area  Score 

ProcessS:  Process  Area  Score 

Products:  Product  Area  Score 

RiskS:  Risk  Area  Score 

2.  Software  Project  Management  Sub  Area  Evaluation  Models 

The  people  main  area  score  (PeopleS)  is  calculated  as  follows: 

e  »  (C*h  T  L  +  OQ  ■+■  PM  •H  SI  +  S) 

(2)  People  Area  Score  - - - - - 
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where: 


C:  Communication  Area  Score 
T:  Teamwork  Area  Score 
L:  Leadership  Area  Score 
OC:  Organizational  Commitment  Area  Score 
PM:  Project  Management  Area  Score 
SI:  Stakeholder  Involvement  Area  Score 
S:  Staffing  and  Hiring  Area  Score 

The  process  main  area  score  (ProcessS)  is  calculated  as  follows: 

„  {RM  +  PMC  +  PPE+SM) 

wProceasArea  Score  ■ - - - - 

4 


where: 


RM:  Requirements  Management  Area  Score 
PMC:  Project  Monitoring  and  Control  Area  Score 
PPE:  Project  Planning  and  Estimation  Area  Score 
SM:  Scope  Management  Area  Score 

The  product  main  area  score  (Products)  is  calculated  as  follows: 

(4)  Pwd  at  Area  Score  =  ^  '“',l 


where: 


CM:  Configuration  Management  Score 
QE:  Quality  Engineering  Score 

The  risk  main  area  score  (RiskS)  is  calculated  as  follows: 

(5)  ittsfc  Ana  Score  =  *  RCJ 


where: 


RA:  Risk  Assessment  Area  Score 
RC:  Risk  Control  Area  Score 

3.  Software  Project  Management  Sub  Area  Evaluation  Models 

The  main  area  scores  are  derived  from  the  sub  area  scores.  The  sub  area 

scores  are  derived  from  participant’s  response  to  the  questionnaire.  For  each 

response  to  a  question  in  the  SPMEI,  there  is  an  associated  score.  The 
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associated  scores  for  each  response  are  provided  in  Appendix  C.  Adding  all  the 
scores  together  in  a  sub  area  provides  an  initial  score  for  that  section  (or  sub 
area). 

It  is  possible  the  initial  score  for  a  sub  area  will  be  a  negative  number  as 
demonstrated  in  Table  15.  The  initial  score  for  each  section  is  made  positive  by 
adding  a  shifting  factor.  This  shifted  score  is  normalized  to  a  scale  of  0  to  10  by 
multiplying  it  with  a  scaling  factor.  Table  16  provides  an  example  of  the  shifting 
factors  and  scaling  factors  as  derived  from  the  values  in  Table  15. 


Table  15.  Example  Scoring  Ranges 


People 

No  of 

Questions 

Lowest 

Score 

Highest 

Score 

Difference 

Communication 

23 

-38 

66 

104 

Teamwork 

30 

-54 

73 

127 

The  steps  for  calculating  the  score  for  a  project  management  area  are 
listed  as  follows: 

1)  Sum  the  scores  for  each  response  in  the  section  together.  This  is 
the  initial  score  for  the  sub  project  management  area. 

2)  Add  the  shifting  factor  to  initial  score.  This  becomes  the  shifted 
initial  score  for  the  sub  project  management  area. 

3)  Multiply  the  shifted  initial  score  with  the  associated  scaling  factor  to 
normalize  the  score  to  a  scale  of  0  to  10.  This  normalized  score  for 
the  sub  project  management  area  can  now  be  fed  into  sub  area 
evaluation  model. 


Table  16. 

Example  Shifting  and  Scaling  Factors 

Project 
Sub  Area 

Management 

Shifting  factor 

Scaling  factor 

Communication 

38 

10/104 

Teamwork 

54 

10/127 
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The  generic  model  to  determine  a  project  management  area  score  is: 


(0  Project  ManagementSub  Area  Score  =  Scaling  Factor* 


PM  At  +  Shifting  Factor ) 


1 


In  Equation  6,  n  is  the  number  of  questions  in  the  section.  PMAi  is  the  sum 
of  the  scores  for  each  response  in  a  section.  For  example,  in  the  communication 
section  of  the  SPMEI,  there  are  twenty-three  questions.  Thus,  n  is  23  for  this  sub 
area  model.  For  the  communication  area  score  in  Equation  7,  the  scaling  factor  is 
10/104  and  the  shifting  factor  is  38.  For  the  complete  details  of  the  SPMEM  refer 
to  Appendix  D. 


(7)  Communication  Area  Score  ■ 


D.  SUMMARY  OF  RESULTS  OF  INITIAL  STUDY 


Sixteen  software  projects  were  surveyed  by  Demir.  The  graph  below 
shows  a  plot  of  project  success  ratings  and  PME  scores.  The  trend  suggests  that 
there  is  a  relationship  between  the  PME  score  and  the  project  success  rating.  At 
first  look,  it  would  appear  that  the  higher  the  PME  score  the  higher  the  project 
success  rating. 


Figure  9.  Project  Success  Ratings  and  PME  Scores 
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In  order  to  understand  the  association  between  the  PME  score  and  the 
project  success  rating  of  a  project,  a  correlation  analysis  was  conducted  by 
Demir.  The  Pearson  product-moment  correlation  coefficient  was  used  to  identify 
the  linear  relationship  between  the  sets  of  calculated  variables.  The  Pearson 
product-moment  correlation  coefficient  between  the  project  success  ratings  and 
the  PME  scores  is  0.68.  This  indicates  a  strong  positive  correlation  between  the 
two  variables.  Demir’s  study  suggests  that  when  project  management 
effectiveness  is  high,  project  success  is  more  likely.  It  was  demonstrated  that  it  is 
possible  to  develop  a  metric  to  measure  the  effectiveness  of  software  project 
management. 

1.  External  Validity 

The  small  sample  size  of  Demir’s  study  is  an  obvious  limitation  and 
reduces  the  external  validity  of  the  study.  It  is  difficult  to  make  generalizations 
about  the  use  of  the  metric  on  other  projects  with  a  sample  size  of  sixteen. 
Additionally,  there  is  only  one  project  that  has  a  lower  project  success  rating  than 
five  and  the  subjects  were  only  from  America  and  Europe.  Increasing  the  size  of 
the  sample  and  the  range  of  success  ratings  should  prove  insightful.  The  goal  of 
the  research  in  this  thesis  was  to  increase  the  sample  size  by  measuring  more 
projects  in  order  to  provide  further  insight  to  the  applicability  and  limitations  of  the 
metric. 
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IV.  METHODS 


A.  INTRODUCTION 

The  development  of  a  tool  that  measures  the  software  project 

management  effectiveness  could  prove  to  be  highly  valuable  to  software  project 

managers.  The  Software  Project  Management  Effectiveness  Metric  is  one  of 
these  tools  that  have  shown  promise.  To  discover  how  promising  the  metric  is, 
the  following  research  questions  were  addressed  in  this  study: 

1 )  Will  improving  a  project’s  PME  score  increase  the  project’s  chance 
of  success? 

(a)  What  is  the  relationship  between  the  PME  score 

(measured)  and  the  project  success  rating  (measured)? 

(b)  What  is  the  relationship  between  the  PME  score 

(measured)  and  the  size  of  the  project  (measured)? 

(c)  What  is  the  relationship  between  an  institution’s  CMMI 
level  (measured  variable)  and  the  PME  metric  (measured 
variable)? 

2)  What  are  software  development  practitioner’s  perceptions  towards 
the  practicality  and  usefulness  of  the  metric? 

(a)  What  are  software  development  practitioner’s 
perceptions  towards  the  manageability,  meaningfulness, 
actionability,  ambiguity,  reliability,  accuracy,  timeliness 
and  predictability  of  the  metric? 

(b)  Will  software  development  practitioners  use  the  metric? 

To  answer  these  questions,  the  research  was  conducted  in  two  phases.  In 
phase  one,  participants  used  the  SPMEI  to  measure  a  project  they  had  worked 
on  and  the  SPMEM  was  used  to  obtain  the  PME  score  for  their  project.  Phase 
two  was  a  chance  for  participants  to  provide  their  feedback  on 
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the  metric  through  the  completion  of  a  short  questionnaire.  The  data  obtained  in 
this  study  was  combined  with  Demir’s  for  analysis.  A  visual  depiction  of  the 
research  method  is  illustrated  in  Figure  10. 


Figure  10.  Research  Method  (Activity  Diagram) 
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B.  SAMPLE/PARTICIPANTS 


1.  Sampling  Plan 

Research  subjects  for  this  type  of  study  are  most  likely  to  be  found 
through  personal  networking,  via  friends  and  colleagues  (Demir,  2008).  Due  to 
the  length  and  content  of  the  SPMEI,  potential  subjects  were  recruited  from  the 
researcher’s  professional  network.  This  was  the  sole  means  of  recruiting  subjects 
for  the  study  and,  as  such,  was  a  sample  of  convenience. 

2.  Description  of  Participants 

In  this  study,  a  combined  data  set  was  obtained  by  joining  Demir’s  data 
(henceforth  referred  to  as  existing  data  set)  and  the  new  data  set,  obtained  by 
the  research  in  this  thesis.  Nine  projects  were  surveyed  to  create  the  new  data 
set  and  the  details  of  these  projects  are  published  in  Table  17.  The  sample 
contains  very  recent  projects  of  varying  durations.  The  software  products 
developed  ranged  from  weapon  systems  software  to  web  applications. 


Table  17.  New  Data  Set  Sample 


Project 

Delivery  Date 

Software  Product 

Duration  (months) 

AA 

2008 

Command  and  Control 

24 

BB 

2010 

Web  Application 

44 

CC 

2010 

Weapon  System 

29 

DD 

2010 

Command  and  Control 

28 

EE 

2011 

Information  and  Data  Management 

28 

FF 

2010 

Entertainment 

NA 

GG 

2010 

Web  Application 

12 

HH 

2010 

Weapon  System 

11 

II 

2010 

Web  Application 

18 

The  combined  data  set  contains  25  projects.  The  duration  of  the  projects 
in  the  sample  can  be  seen  in  Figure  12.  The  average  project  duration  was  20 
months.  The  combined  data  set  contains  projects  mainly  from  the  last  six  years. 
The  time  frame  for  the  projects  is  displayed  in  Figure  13. 
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Figure  1 1  presents  the  combined  sample  in  terms  of  the  average  number 
of  people  involved.  The  projects  are  divided  into  four  sizes:  small,  medium,  large 
and  very  large.  More  than  half  of  the  projects  in  the  combined  sample  are  small 
size  projects.  One  quarter  is  medium  size  and  the  remaining  larger  projects  make 
up  the  rest  of  the  sample. 


Very  Large 

Large  (20-100  (100+ People) 


Figure  1 1 .  Project  Size  in  Terms  of  Average  People  Involved 
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Duration  (Months) 


Figure  12.  Project  Duration 

7 


4 


2  2  2 

1  1 

3 

1 

i 

0  0 

LL 

LL 

Lt 

1977  1983  1995  2000  2001  2002  2003  2004  2005.  2006  2007  2008  2009  2010  2011 

Year  of  Project  Delivery 


Figure  13.  Project  Delivery 
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C.  MEASUREMENT  INSTRUMENTS 

1.  Phase  1:  Software  Project  Management  Evaluation  Instrument 

The  SPMEI  was  used  for  the  first  phase  of  the  study  and  is  included  in 
Appendix  B  of  this  thesis.  The  SPMEI  is  scored  by  using  the  tables  in  Appendix 
C. 


2.  Phase  2:  Metric  Feedback  Instrument 

The  second  phase  of  the  study  used  the  Metric  Feedback  Instrument  that 
was  specifically  developed  for  this  research.  The  objective  of  this  instrument  was 
to  obtain  the  subject’s  opinion  on  the  usefulness  of  the  metric.  The  instrument 
was  created  by  using  the  eight  attributes  of  good  metrics  as  published  by  Brotby 
(2009).  The  instrument  subjectively  measures  the  manageability, 
meaningfulness,  actionability,  ambiguity,  reliability,  accuracy,  timeliness  and 
predictability  of  the  metric.  A  description  of  each  attribute  is  printed  in  Table  18. 
Each  one  is  subjectively  assessed  on  a  scale  of  one  to  ten.  The  participants  are 
also  asked  to  provide  their  own  comments  on  each  attribute  of  the  metric  and  are 
queried  to  see  if  they  would  use  the  PME  metric  on  their  next  software  project.  A 
copy  of  the  instrument  is  provided  in  Appendix  E. 
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Table  18.  Attributes  of  Good  Metrics  (From:  Brotby,  2009) 


Attribute 

Description 

Manageability 

A  metric’s  information  should  be  available  and  concise 

Meaningful 

A  metric  must  be  understandable  and  relevant  to  the  recipient 
and  provide  a  basis  for  decisions 

Actionable 

Useful  metrics  information  makes  it  clear  what  response  is 
needed,  as  a  compass  makes  it  clear  whether  to  turn  left  or  right 
or  stay  on  course 

Ambiguity 

Information  from  metrics  can  have  a  number  of  meanings  and 
may  be  misleading,  of  little  use,  or  downright  dangerous 

Reliability 

The  ability  to  trust  the  “instrument”  is  conditioned  on  the 
reliability  of  the  measurement 

Accuracy 

A  reasonable  and  known  degree  of  a  metric’s  accuracy  is 
essential.  The  compass  showing  north  when  we  are  going 
south  can  be  fatal 

Timely 

Measures  that  warn  of  a  disaster  after  it  has  happened  are  not 
useful 

Predictive 

Some  metrics  information  will  signal  impending  problems  much 
as  a  drop  in  oil  pressure  is  the  harbinger  of  engine  failure 

3.  Validity  and  Reliability 

The  effectiveness  of  metric  is  the  extent  to  which  it  provides  information 
that  meets  the  previously  defined  criteria  for  the  recipient.  The  instrument 
produces  a  quantitative  score  out  of  80  on  the  effectiveness  of  the  metric  (not  to 
be  confused  with  the  software  project  management  effectiveness).  The 
instrument  will  also  provide  qualitative  responses  on  the  attributes  of  the  metric. 
The  metric  feedback  instrument  provided  a  reasonably  good  and  consistent 
measure  of  the  metric’s  effectiveness. 

D.  DATA  COLLECTION  PROCEDURES 

1.  Phase  1:  Software  Project  Management  Evaluation  Instrument 

Potential  subjects  were  contacted  directly  through  the  previously 
mentioned  networking  approach  and  informed  of  the  study.  If  they  were 
interested  in  participating,  they  were  emailed  a  link  to  the  SPMEI,  which  was 
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hosted  online  by  SurveyMonkey.  The  participant  connections  to  the  online 
SPMEI  were  protected  by  VeriSign  certificate  Version  3  with  128  bit  encryption. 
This  provided  assurance  that  participant  responses  were  communicated  securely 
to  and  from  the  SurveyMonkey  servers. 

Risk.  Due  to  the  nature  of  the  data  obtained  from  the  questionnaire,  the 
risk  to  the  subjects  was  deemed  to  be  very  low.  A  breach  of  the  subject’s 
confidentiality  may  result  in  some  embarrassment  for  the  subject. 

Consent.  It  was  the  investigator’s  responsibility  to  obtain  informed 
consent  from  the  subjects  before  they  commenced  the  survey.  A  waiver  from  the 
requirement  to  document  the  informed  consent  was  obtained  from  the  IRB. 

Data.  The  subject’s  data  was  retrieved  from  the  SurveyMonkey  servers 
and  stored  on  NPS  servers  in  order  to  conduct  the  research  analysis.  The 
researchers  will  ensure  that  the  subject’s  confidentiality  is  maintained.  No 
information  was  made  publicly  accessible  that  could  identify  the  participants. 

2.  Phase  2:  Metric  Feedback  Instrument 

After  the  data  was  collected  from  the  SPMEI,  a  metric  was  produced  using 
the  SPMEM  for  each  project.  The  participant  was  then  provided  with  a  report  on 
their  project’s  PME  scores.  An  example  of  this  report  is  provided  in  Appendix  F. 
The  report  maintains  the  subject’s  confidentiality.  The  instrument  was  distributed 
and  data  was  collected  in  the  exact  same  way  as  phase  1 . 

E.  DATA  ANALYSIS 

1.  Phase  1:  Software  Project  Management  Evaluation  Instrument 

Before  any  analysis  was  conducted,  the  PME  scores  for  each  project  in 
the  new  data  set  was  calculated  and  subsequently  combined  with  the  PME 
scores  from  the  existing  data  set.  The  subjects  recorded  a  project  success  rating 
at  the  start  of  the  questionnaire  and  then  again  at  the  end.  The  average  project 

success  rating  was  used  for  the  correlation  analysis. 
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In  order  to  determine  the  relationship  between  the  PME  score  and  the 
project  success  rating,  a  correlation  analysis  was  conducted.  The  Pearson 
product-moment  correlation  coefficient  (PMCC)  was  used  to  identify  the  linear 
relationship  between  the  two  measured  variables.  This  analysis  also  allowed  the 
researcher  to  test  the  hypothesis  that  the  PME  score  positively  correlates  to  the 
project  success  rating. 

The  calculated  PMCC,  or  r  for  this  sample,  will  always  lie  between  -1  and 
1.  The  polarity  of  r  indicates  the  direction  of  the  linear  relation.  In  a  positive 
correlation,  when  one  variable  goes  up  the  other  variable  goes  up  as  well.  In  a 
negative  correlation,  when  one  variable  goes  up,  the  other  variable  goes  down. 

The  absolute  value  of  r  indicates  the  strength  of  the  linear  relationship. 
The  higher  the  value  of  r,  the  stronger  the  linear  relationship  between  the 
variables  is.  When  the  absolute  value  of  r  is  1,  this  indicates  that  there  is  a 
perfect  correlation  between  the  two  variables.  Perfect  relationships  are  rarely 
observed  in  social  studies.  In  social  studies,  as  a  rule  of  thumb,  when  the 
absolute  value  of  r  is  greater  than  0.5,  then  it  may  be  assumed  that  there  is 
strong  correlation  between  the  variables  (Demir,  2008).  When  r  is  below  0.5,  the 
linear  relationship  between  the  variables  is  weak.  A  summary  of  the  data  analysis 
is  described  in  Table  19. 
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Table  19.  Will  improving  a  project’s  PME  score  increase  the  project’s  chance  of 

success? 


Research  Question _ Analysis _ Data  Collected 


What  is  the  relationship  Calculated  the  PME  score  for  25  Project  Success  Ratings 

between  the  PME  score  the  25  projects.  Calculated  the  and  SPMEI  data  on  25 

(measured)  and  the  project  PPMC  between  the  PME  projects 

success  rating  (measured)?  score  and  Project  Success 

Rating _ 

What  is  the  relationship  Calculated  the  PPMC  between  Obtained  data  on  the  size  of 

between  the  PME  score  PME  score  and  the  size  of  the  the  project  in  terms  of  people 

(measured)  and  the  size  of  the  project  involved 

project  (measured)? 

What  is  the  relationship  Calculated  the  PPMC  between  Obtained  CMMI  levels  for  9 

between  an  institution’s  CMMI  the  PPMC  between  the  CMMI  projects 

level  (measured  variable)  and  level  and  PME  score 

the  PME  metric  (measured 

variable)? 


2.  Phase  2:  Metric  Feedback 

The  opinion  data  collected  was  categorized  in  terms  of  research  questions 
and  emergent  themes.  A  coding  method  was  used  to  organize  data  into  a  limited 
number  of  themes  and  issues  around  the  questions.  Quotations  were  then 
selected  that  illuminated  the  themes  and  concepts. 

Quantitative  data  analysis  was  also  performed  on  the  subject  scores  of  the 
metric  attributes.  The  results  of  the  survey  were  analyzed  using  descriptive 
statistics.  The  range,  mean  and  standard  deviation  were  obtained  for  each  of  the 
attributes.  This  statistics  were  also  obtained  for  the  total  score  for  all  of  the  metric 
attributes. 
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V.  RESULTS 


A.  PHASE  ONE  RESULTS 

1.  Project  Success  Rating  Results 

Participants  in  phase  one  subjectively  reported  the  success  of  their  project 
on  a  scale  of  0  to  10  (0  for  a  complete  failure  and  10  for  a  complete  success). 
Figure  14  is  a  histogram  of  the  rounded  project  success  ratings  recorded  for  the 
combined  data  set.  The  mean  rating,  for  the  25  projects  in  the  combined  data 
set,  was  7.2.  The  mode  of  the  project  success  ratings  was  7.  The  smallest 
success  rating  was  2.5  and  the  highest  was  10.  If  a  score  of  5  or  above  is 
considered  to  be  a  success,  then  88%  of  projects  were  rated  as  successful  by 
the  participants.  Projects  with  scores  of  0,  1  or  2  were  not  represented  in  the 
sample.  There  were  no  projects  sampled  that  were  cancelled.  The  external 
validity  of  the  sample  would  be  increased  if  the  lower  range  of  project  success 
scores  was  increased  in  the  sample. 

I  I 
■■■Mill 

123456789  10 

Project  Success  Rating 

Figure  14.  Project  Success  Rating  Histogram 
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Figure  15  presents  the  mean  performance  statistics  of  the  projects 
(combined  data  set).  On  average,  the  projects  delivered  97%  of  the  required 
functionality,  were  31%  behind  schedule  estimated  and  were  23%  over  budget.  It 
should  be  noted  that  not  all  projects  reported  their  budget.  The  complete  project 
statistics  are  contained  in  Table  20.  In  some  cases,  there  were  significant  cost 
and  schedule  overruns;  however,  the  projects  were  still  rated  as  a  success.  The 
project  success  rating  is  based  on  the  eye  of  the  beholder.  If  cost  and  schedule 
were  not  considered  a  priority,  but  functionality  was  crucial  to  success,  then  a 
project  can  still  be  rated  as  successful. 


100% 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 

Delivered  Schedule  Cost  Overrun  Percentage  of 
Functionality  Overrun  Successful 

Projects 


Figure  15.  Average  Project  Performance  Statistics 
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Table  20.  General  Project  Statistics 


Project 

Delivery 

Software  Category 

Duration 

KSLOC 

Delivered 

Schedule 

Cost 

Project 

Date 

Functionality 

Overrun 

Overrun 

Success 

Rating 

A 

2006 

1  nformation  and  Data 

Management 

10 

100 

100% 

43% 

50% 

6 

B 

2006 

Embedded  system 

6 

95% 

0% 

9 

C 

2009 

Embedded  system 

17 

100 

100% 

42% 

29% 

5 

D 

2006 

Embedded  system 

4 

30 

100% 

0% 

0% 

7 

E 

1995 

Supply  Chain  Management 

24 

70% 

167% 

3 

F 

2008 

Customer  Service 

12 

95% 

20% 

7 

G 

1983 

Command  and  Control 

10 

16 

70% 

0% 

8 

H 

2005 

Command  and  Control 

24 

10 

90% 

0% 

0% 

7 

1 

1983 

Records  Management 

36 

150% 

0% 

10 

J 

1977 

1  nternet  Utilities  and  Applications 

12 

100% 

0% 

0% 

10 

K 

2008 

24 

215 

100% 

0% 

9 

L 

2005 

Weapon  System 

30 

440 

100% 

25% 

7 

M 

2006 

Security  Applications 

7 

115 

95% 

17% 

8 

N 

2002 

Weapon  System 

24 

98% 

0% 

0% 

9 

O 

2007 

Security  Applications 

30 

80% 

25% 

7 

P 

1995 

Scientific  Service  Delivery 

14 

100% 

17% 

9 

AA 

2008 

Command  and  Control 

24 

75% 

-20% 

8 

BB 

2010 

Web  Application 

44 

2000 

150% 

144% 

50% 

3.5 

CC 

2010 

Weapon  System 

29 

100% 

61% 

70% 

5.5 

DD 

2010 

Command  and  Control 

28 

85 

85% 

22% 

7 

EE 

2011 

1  nformation  and  Data 

Management 

28 

80% 

133% 

6.5 

FF 

2010 

Entertainment 

100% 

6 

GG 

2010 

Web  Application 

12 

25 

100% 

0% 

8% 

9 

HH 

2010 

Weapon  System 

11 

230 

100% 

-8% 

10 
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Project 

Delivery 

Date 

Software  Category 

Duration 

KSLOC 

Delivered 

Functionality 

Schedule 

Overrun 

Cost 

Overrun 

Project 

Success 

Rating 

II 

2010 

Web  Application 

18 

20 

80% 

50% 

2.5 

Min 

1977 

4 

10 

70% 

-20% 

0% 

2.5 

max 

2011 

44 

2000 

150% 

167% 

70% 

10.0 

mode 

2010 

24 

100 

100% 

0% 

0% 

7 

median 

21 

100 

1 

0.17 

0.08 

7 

range 

34 

40 

1990 

80% 

187% 

70% 

7.5 

mean 

20 

260 

97% 

31% 

23% 

7.2 
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2. 


Software  Project  Management  Evaluation  Model  Results 


This  section  contains  various  tables  showing  the  sub  area  scores,  main 
area  scores  and  PME  scores  for  all  of  the  projects  in  the  combined  data  set. 
Descriptive  statistics  are  also  contained  in  the  tables.  Table  22  presents  the 
People  sub  area  scores  calculated  using  the  SPMEM.  Table  23  shows  the 
Process  sub  area  scores  and  Table  24  displays  the  Product  sub  area  scores. 
Lastly,  Table  25  shows  the  Risk  sub  area  scores  and  Table  26  contains  the  PME 
scores  and  project  success  rating. 

The  People,  Process  and  Product  scores  all  had  similar  mean  scores  with 
6.6,  6.2  and  6.6  respectively.  On  average,  the  Risk  area  score  was  measured  as 
the  lowest  performer  with  a  mean  of  5.6.  This  indicated  that  the  projects  in  the 
sample  all  needed  to  work  on  improving  their  risk  management  practices.  The 
range  of  main  area  scores  and  the  PME  score  are  all  close  to  each  other.  Two  of 
the  projects  obtained  a  score  of  10  in  different  sub  areas,  indicating  that  perfect 
scores  are  possible.  The  minimum  main  area  score  was  the  risk  area,  with  2.5 
and  the  maximum  was  the  product  area  with  a  score  of  9.7. 

The  lowest  PME  score  calculated  was  3.1,  while  the  highest  was  8.8.  The 
mean  of  the  PME  scores  was  6.3.  Figure  16  is  a  histogram  of  the  rounded  PME 
scores,  which  has  a  mode  of  6.  It  is  important  to  highlight  that  every  project  in  the 
combined  data  set  with  a  PME  score  of  6  or  above  was  successful.  In  other 
words,  every  project  with  a  PME  score  of  6  or  greater  had  a  project  success 
rating  of  5  or  greater.  Table  21  shows  the  average  project  success  rating  for 
three  different  brackets  of  PME  scores.  It  shows  a  distinct  positive  increase  in 
success  ratings  as  you  move  up  through  the  brackets. 


75 


Table  21 .  PME  Score  Brackets 


PME 

Average  Project  Success  Rating 

>=5  and  <6 

6 

>=6  and  <7 

7.8 

>7 

8.5 

Table  22.  Results  of  People  Area  Scores 


Project 

C 

T 

L 

OC 

PM 

SI 

S 

People 

A 

6.5 

6.1 

5.9 

5.4 

7.1 

6.8 

4.4 

6.0 

B 

7.2 

7.8 

8.4 

7.3 

7.9 

6.6 

7.3 

7.5 

C 

6.4 

6.1 

6.2 

7.6 

5.2 

7.7 

5.9 

6.4 

D 

7.1 

7.1 

6.6 

6.7 

8.1 

7.7 

5.9 

7.0 

E 

6.3 

7.1 

5.0 

8.1 

6.7 

5.7 

6.1 

6.4 

F 

6.1 

6.4 

7.9 

6.4 

7.7 

3.4 

6.3 

6.3 

G 

5.0 

5.9 

5.9 

5.7 

6.6 

5.4 

6.6 

5.9 

H 

6.2 

5.7 

5.9 

5.3 

6.4 

7.2 

4.6 

5.9 

1 

7.8 

8.0 

6.9 

7.5 

9.1 

8.9 

7.2 

7.9 

J 

8.8 

7.8 

7.9 

8.0 

8.6 

6.8 

6.8 

7.8 

K 

6.8 

6.9 

7.4 

6.9 

7.4 

6.0 

6.5 

6.8 

L 

5.1 

5.7 

6.6 

6.5 

6.3 

4.2 

5.7 

5.7 

M 

6.3 

7.6 

8.1 

7.4 

7.8 

6.3 

6.4 

7.1 

N 

9.2 

7.6 

9.1 

6.6 

9.2 

7.5 

5.8 

7.9 

O 

6.3 

6.5 

6.3 

7.9 

8.1 

6.5 

7.9 

7.1 

P 

9.0 

9.6 

9.1 

10.0 

9.6 

8.3 

9.8 

9.4 

AA 

7.0 

6.4 

7.2 

6.7 

7.5 

6.8 

7.1 

7.0 

BB 

5.7 

5.5 

7.2 

6.0 

6.7 

3.2 

5.7 

5.7 

CC 

4.9 

6.1 

4.3 

7.1 

6.3 

4.4 

6.1 

5.6 

DD 

7.2 

5.6 

7.1 

6.2 

7.0 

6.8 

6.6 

6.6 

EE 

4.4 

4.2 

5.1 

4.4 

5.6 

4.7 

4.6 

4.7 

FF 

8.5 

8.3 

7.9 

8.4 

8.0 

6.3 

8.0 

7.9 

GG 

6.2 

5.7 

5.6 

5.7 

6.9 

5.1 

5.9 

5.9 

HH 

6.6 

7.4 

6.9 

7.1 

7.2 

6.3 

6.7 

6.9 

II 

3.0 

2.4 

2.1 

3.5 

2.5 

5.4 

3.1 

3.1 

Min 

3.0 

2.4 

2.1 

3.5 

2.5 

3.2 

3.1 

3.1 

Max 

9.2 

9.6 

9.1 

10.0 

9.6 

8.9 

9.8 

9.4 

Range 

6.3 

7.2 

7.1 

6.5 

7.1 

5.7 

6.7 

6.2 

Mean 

6.5 

6.5 

6.7 

6.7 

7.2 

6.2 

6.3 

6.6 

Standard  Deviation 

1.46 

1.44 

1.56 

1.35 

1.46 

1.44 

1.33 

1.23 

Variation 

2.12 

2.09 

2.44 

1.83 

2.13 

2.07 

1.77 

1.50 
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Table  23.  Results  of  Process  Area  Scores 


Project 

RM 

PMC 

PPE 

SM 

PROCESS 

A 

7.0 

7.6 

6.4 

6.9 

7.0 

B 

7.1 

5.5 

6.8 

7.0 

6.6 

C 

7.2 

6.2 

5.6 

6.1 

6.3 

D 

5.4 

6.9 

6.7 

5.9 

6.2 

E 

3.8 

5.2 

7.1 

4.2 

5.1 

F 

4.8 

6.6 

4.9 

4.9 

5.3 

G 

7.0 

7.0 

6.3 

6.1 

6.6 

H 

5.3 

5.1 

6.2 

3.7 

5.1 

1 

7.3 

7.9 

7.9 

6.9 

7.5 

J 

6.8 

7.0 

7.4 

5.8 

6.7 

K 

7.2 

6.2 

6.6 

6.3 

6.6 

L 

6.1 

5.6 

5.2 

3.4 

5.1 

M 

6.1 

6.4 

5.9 

5.8 

6.0 

N 

8.0 

8.1 

7.3 

7.7 

7.8 

O 

9.2 

5.8 

7.1 

5.6 

6.9 

P 

9.7 

8.1 

8.7 

7.9 

8.6 

AA 

5.4 

6.6 

6.6 

5.8 

6.1 

BB 

5.1 

5.6 

5.9 

3.1 

4.9 

CC 

7.5 

6.5 

6.7 

6.5 

6.8 

DD 

5.1 

6.4 

6.5 

5.9 

6.0 

EE 

4.4 

6.7 

5.3 

4.9 

5.4 

FF 

6.9 

6.7 

7.2 

7.5 

7.1 

GG 

6.7 

6.6 

5.8 

5.4 

6.1 

HH 

7.1 

7.6 

6.6 

6.9 

7.0 

II 

3.8 

1.9 

2.5 

2.1 

2.6 

Min 

3.8 

1.9 

2.5 

2.1 

2.6 

Max 

9.7 

8.1 

8.7 

7.9 

8.6 

Range 

5.9 

6.3 

6.3 

5.8 

6.0 

Mean 

6.4 

6.4 

6.4 

5.7 

6.2 

Standard  Deviation 

1.49 

1.27 

1.17 

1.48 

1.19 

Variation 

2.22 

1.61 

1.38 

2.18 

1.41 
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Table  24.  Results  of  Product  Area  Scores 


Project 

CM 

QE 

PRODUCT 

A 

8.7 

7.1 

7.9 

B 

7.2 

6.9 

7.0 

C 

4.5 

7.8 

6.2 

D 

2.2 

5.6 

3.9 

E 

2.2 

7.1 

4.6 

F 

4.0 

7.2 

5.6 

G 

8.2 

7.5 

7.8 

H 

5.5 

5.5 

5.5 

1 

8.5 

8.1 

8.3 

J 

5.5 

8.4 

6.9 

K 

8.7 

6.9 

7.8 

L 

8.0 

6.2 

7.1 

M 

5.0 

5.4 

5.2 

N 

7.2 

7.3 

7.2 

O 

8.2 

6.8 

7.5 

P 

9.3 

9.7 

9.5 

AA 

4.5 

7.1 

5.8 

BB 

9.5 

6.3 

7.9 

CC 

10.0 

9.5 

9.7 

DD 

4.0 

3.5 

3.8 

EE 

5.2 

4.7 

4.9 

FF 

5.7 

7.6 

6.7 

GG 

8.7 

7.2 

7.9 

HH 

7.8 

5.7 

6.8 

II 

7.2 

1.8 

4.5 

Min 

2.2 

1.8 

3.8 

Max 

10.0 

9.7 

9.7 

Range 

7.8 

7.8 

6.0 

Mean 

6.6 

6.7 

6.6 

Standard  Deviation 

2.26 

1.69 

1.61 

Variation 

5.12 

2.87 

2.60 
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Table  25.  Results  of  Risk  Area  Scores 


Project 

RA 

RC 

RISK 

A 

6.4 

6.3 

6.3 

B 

5.6 

5.9 

5.8 

C 

5.5 

4.4 

5.0 

D 

5.5 

5.7 

5.6 

E 

3.7 

3.7 

3.7 

F 

5.6 

5.4 

5.5 

G 

6.8 

5.9 

6.4 

H 

5.0 

6.3 

5.7 

1 

7.6 

8.1 

7.9 

J 

4.9 

6.1 

5.5 

K 

6.4 

4.6 

5.5 

L 

3.8 

3.7 

3.7 

M 

6.2 

5.0 

5.6 

N 

8.1 

8.0 

8.0 

O 

6.2 

5.0 

5.6 

P 

8.5 

5.6 

7.0 

AA 

5.6 

6.1 

5.9 

BB 

2.6 

5.2 

3.9 

CC 

6.0 

6.1 

6.1 

DD 

4.8 

6.1 

5.5 

EE 

6.1 

5.7 

5.9 

FF 

6.3 

6.7 

6.5 

GG 

4.9 

4.4 

4.7 

HH 

6.0 

6.5 

6.3 

II 

2.8 

2.2 

2.5 

Min 

2.6 

2.2 

2.5 

Max 

8.5 

8.1 

8.0 

Range 

5.9 

5.9 

5.5 

Mean 

5.6 

5.6 

5.6 

Standard  Deviation 

1.43 

1.28 

1.24 

Variation 

2.04 

1.64 

1.53 
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Table  26.  Main  Area  Scores  and  PME  scores 


Project 

PEOPLE 

PROCESS 

PRODUCT 

RISK 

PME 

Success  Rating 

A 

6.0 

7.0 

7.9 

6.3 

6.7 

6.0 

B 

7.5 

6.6 

7.0 

5.8 

6.8 

9.0 

C 

6.4 

6.3 

6.2 

5.0 

6.1 

5.0 

D 

7.0 

6.2 

3.9 

5.6 

5.9 

7.0 

E 

6.4 

5.1 

4.6 

3.7 

5.2 

3.0 

F 

6.3 

5.3 

5.6 

5.5 

5.7 

7.0 

G 

5.9 

6.6 

7.8 

6.4 

6.6 

8.0 

H 

5.9 

5.1 

5.5 

5.7 

5.5 

7.0 

1 

7.9 

7.5 

8.3 

7.9 

7.9 

10.0 

J 

7.8 

6.7 

6.9 

5.5 

6.9 

10.0 

K 

6.8 

6.6 

7.8 

5.5 

6.7 

9.0 

L 

5.7 

5.1 

7.1 

3.7 

5.5 

7.0 

M 

7.1 

6.0 

5.2 

5.6 

6.1 

8.0 

N 

7.9 

7.8 

7.2 

8.0 

7.8 

9.0 

O 

7.1 

6.9 

7.5 

5.6 

6.9 

7.0 

P 

9.4 

8.6 

9.5 

7.0 

8.8 

9.0 

AA 

7.0 

6.1 

5.8 

5.9 

6.3 

8.0 

BB 

5.7 

4.9 

7.9 

3.9 

5.6 

3.5 

CC 

5.6 

6.8 

9.7 

6.1 

6.9 

5.5 

DD 

6.6 

6.0 

3.8 

5.5 

5.7 

7.0 

EE 

4.7 

5.4 

4.9 

5.9 

5.2 

6.5 

FF 

7.9 

7.1 

6.7 

6.5 

7.2 

6.0 

GG 

5.9 

6.1 

7.9 

4.7 

6.2 

9.0 

HH 

6.9 

7.0 

6.8 

6.3 

6.8 

10.0 

II 

3.1 

2.6 

4.5 

2.5 

3.1 

2.5 

Min 

3.1 

2.6 

3.8 

2.5 

3.1 

2.5 

Max 

9.4 

8.6 

9.7 

8.0 

8.8 

10.0 

Range 

6.2 

6.0 

6.0 

5.5 

5.6 

7.5 

Mean 

6.6 

6.2 

6.6 

5.6 

6.3 

7.2 

Standard 

Deviation 

1.23 

1.19 

1.61 

1.24 

1.09 

2.11 

Variation 

1.50 

1.41 

2.60 

1.53 

1.19 

4.43 
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Figure  16.  Rounded  PME  Scores  Histogram 


3.  PME  Score  and  Project  Success  Rating  Relationship 

Figure  17  shows  a  plot  of  the  project  success  rating  and  the  PME  score 
(sorted  by  the  lowest  success  rating  to  the  highest).  At  a  glance,  it  would  seem 
that  the  higher  the  project  success  rating  the  higher  the  PME  score.  An 
interesting  phenomenon  appears  to  be  present  as  well.  When  the  project 
success  rating  is  6  or  below,  the  PME  score  is  greater  than  the  success  rating. 
When  the  project  success  rating  is  above  6,  the  scores  invert  and  the  PME  score 
is  less  than  the  success  rating.  It  is  difficult  to  make  assertions  about  this  trend 
with  the  current  sample  size. 
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Figure  17.  PME  Score  and  Project  Success  Rating  (lowest  success  to  highest) 


a.  Hypothesis  Testing 

The  results  of  the  PMCC  analysis  are  contained  in  Tables  27  and 
28.  The  project  success  rating  was  graphed  against  the  PME  score  in  Figure  18. 
A  quick  look  at  this  plot  shows  the  likely  existence  of  linear  relationship  between 
the  PME  score  and  the  project  success  rating.  The  correlation  between  these  two 
variables  was  found  to  be  0.68,  which  confirms  the  hypothesis:  The  success  of  a 
software  project  positively  correlates  to  its  software  project  management 
effectiveness  metric  score. 
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Figure  18.  PME  Score  vs.  Project  Success  Rating 
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Table  27.  PMCC  Between  Sub  Area  Scores 


C  T  L 

OC 

PM 

SI 

S 

RM 

PMC 

PPE 

SM 

CM 

QE 

RA 

RC 

c 

*  0.86  0.82 

0.71 

0.86 

0.64 

0.66 

0.53 

0.67 

0.81 

0.72 

-0.09 

0.52 

0.56 

0.62 

T 

*  0.80 

0.89 

0.89 

0.51 

0.81 

0.62 

0.68 

0.87 

0.75 

0.02 

0.68 

0.62 

0.53 

L 

* 

0.62 

0.84 

0.31 

0.67 

0.47 

0.64 

0.63 

0.60 

-0.03 

0.43 

0.50 

0.53 

oc 

* 

0.70 

0.40 

0.87 

0.60 

0.47 

0.78 

0.58 

-0.01 

0.70 

0.43 

0.23 

PM 

* 

0.45 

0.74 

0.59 

0.77 

0.86 

0.71 

0.06 

0.60 

0.66 

0.68 

SI 

* 

0.35 

0.45 

0.37 

0.57 

0.57 

-0.14 

0.18 

0.57 

0.45 

S 

* 

0.65 

0.53 

0.76 

0.64 

0.12 

0.61 

0.53 

0.33 

RM 

* 

0.56 

0.63 

0.72 

0.58 

0.65 

0.71 

0.39 

PMC 

* 

0.74 

0.80 

0.13 

0.62 

0.77 

0.76 

PPE 

* 

0.74 

0.10 

0.70 

0.64 

0.65 

SM 

* 

0.16 

0.58 

0.86 

0.69 

CM 

* 

0.31 

0.22 

0.10 

QE 

* 

0.53 

0.39 

RA 

* 

0.67 

RC 

* 
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Table  28.  PMCC  Results  for  Main  Area  Scores,  PME  Score,  Success  Rating,  CMMI  and  Staff  Size 


PEOPLE 

PROCESS 

PRODUCT 

RISK 

PME 

Success 

CMMI 

Staff  Size 

c 

0.93 

0.77 

0.21 

0.65 

0.78 

0.58 

0.37 

0.17 

T 

0.96 

0.82 

0.37 

0.63 

0.85 

0.58 

0.66 

0.22 

L 

0.85 

0.66 

0.21 

0.57 

0.70 

0.62 

0.38 

0.03 

oc 

0.86 

0.69 

0.36 

0.37 

0.72 

0.33 

0.56 

0.15 

PM 

0.92 

0.82 

0.36 

0.73 

0.85 

0.69 

0.47 

0.28 

SI 

0.61 

0.56 

-0.01 

0.56 

0.51 

0.43 

-0.14 

0.37 

S 

0.85 

0.73 

0.41 

0.48 

0.76 

0.50 

0.43 

0.16 

RM 

0.65 

0.84 

0.75 

0.61 

0.86 

0.55 

0.67 

0.15 

PMC 

0.69 

0.87 

0.42 

0.84 

0.82 

0.66 

0.53 

0.23 

PPE 

0.88 

0.87 

0.44 

0.70 

0.87 

0.53 

0.46 

0.27 

SM 

0.76 

0.93 

0.42 

0.85 

0.87 

0.62 

0.54 

0.14 

CM 

-0.02 

0.29 

0.87 

0.18 

0.38 

0.21 

0.35 

0.18 

QE 

0.62 

0.72 

0.75 

0.51 

0.78 

0.35 

0.47 

0.20 

RA 

0.65 

0.85 

0.43 

0.92 

0.82 

0.64 

0.47 

0.27 

RC 

0.57 

0.70 

0.27 

0.90 

0.69 

0.58 

0.48 

0.41 

PEOPLE 

* 

0.84 

0.31 

0.67 

0.86 

0.63 

0.46 

0.23 

PROCESS 

* 

0.58 

0.85 

0.97 

0.67 

0.60 

0.22 

PRODUCT 

* 

0.39 

0.68 

0.33 

0.49 

0.23 

RISK 

* 

0.83 

0.67 

0.51 

0.37 

PME 

* 

0.68 

0.62 

0.29 
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4. 


PME  Score  and  Project  Size  Relationship 


The  correlation,  r,  between  the  PME  score  and  the  average  project  staff  size  was 
0.29.  This  indicates  that  there  is  not  a  linear  relationship  between  the  two  variables. 
This  inference  can  also  be  obtained  from  observing  the  plot  in  Figure  19.  The  graph  in 
Figure  19  excludes  the  project  that  contained  an  average  of  300  project  staff  in  order  to 
focus  on  the  more  concentrated  data  cluster. 

30 
25 
20 

Average  Project  .. 

Staff  Size 

10 
5 
0 

0.0  2.0  4.0  6.0  8.0  10.0 

PME  Score 

Figure  19.  PME  Score  vs.  Average  Project  Staff  Size 
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5.  PME  Score  and  CMMI  Level  Relationship 

The  correlation,  r,  between  the  PME  score  and  a  project’s  CMMI  level  was  0.62. 
This  indicated  that  there  is  a  possible  linear  relationship  between  the  two  variables.  This 
result  is  also  visually  represented  in  Figure  20.  This  sample  size  only  contained  nine 
projects,  which  makes  it  harder  to  draw  solid  conclusions  about  this  relationship. 
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Figure  20.  PME  Score  vs.  Project  CMMI  Level 


6.  Other  Correlation  Analysis  Results 

The  correlation  between  the  main  area  scores  and  the  PME  scores  were  all 
strong.  The  correlation  between  the  process  area  score  and  the  PME  score  in  particular 
needs  to  be  highlighted,  as  it  is  incredibly  strong  (r=0.97).  This  means  that  it  could  be 
possible  to  predict  the  PME  score  based  on  the  process  area  score  alone.  This  does 
not  indicate  that  only  achieving  a  high  process  score  alone  will  give  a  high  PME  score 
because  people,  product  and  risk  all  contribute  to  the  score. 

The  correlation  between  the  product  score  and  the  project  success  rating  was 
0.33.  The  other  three  main  areas  all  had  strong  correlations  with  project  success 
(r=~0.65). 

The  configuration  management  score  had  a  poor  correlation  with  success  at  0.21 
and  quality  engineering  was  similar  at  0.35.  Organizational  commitment  had  one  of  the 
lowest  correlations  with  success  (r=0.33).  The  project  manager  score  had  the  highest 
correlation  with  the  project  success  rating  (r=0.69).  Risk  assessment  and  project 
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monitoring  and  control  also  had  high  correlations  with  success  (r=0.64  and  0.66 
respectively).  Improving  these  scores  would  suggest  an  increased  likelihood  of  success. 

B.  PHASE  2  RESULTS 

The  participants  in  phase  one  were  all  provided  with  their  respective  PME 
scores.  After  reviewing  their  PME  scores,  eight  of  the  original  participants  provided 
feedback,  using  the  Metric  Feedback  Instrument.  The  quantitative  results  are  displayed 
in  Table  29  and  the  qualitative  responses  can  be  examined  in  Appendix  G.  The  average 
effectiveness  score  of  the  metric  was  found  to  be  59  out  of  80  (SD=1 1 .9).  The  individual 
scores  for  each  response  are  presented  graphically  in  Figure  21 . 

1.  Manageable 

For  manageability,  the  metric  scored  a  mean  of  7.  But  due  to  the  large  range,  6, 
it  would  appear  that  opinions  were  quite  divided  over  the  manageability  of  the  metric. 
The  lowest  score  was  4  and  highest  was  10.  No  comments  were  provided  by  the 
participants  on  the  metric’s  manageability. 

2.  Meaningful 

The  metric  scored  high  for  its  meaningfulness  (M=7,  SD=1.7).  It  could  be  said 
that  opinions  were  quite  consistent  over  the  meaningfulness  of  the  metric.  Opinions 
were  generally  positive,  as  echoed  by  one  participant,  “The  survey  seemed  to  translate 
well  into  scores  I  could  relate  to.”  Another  said,  “It  clearly  defines  the  areas  of  good 
performance  and  the  areas  of  concern.”  However,  another  subject  quoted,  “The  metric 
is  meaningless  without  other  data  to  support  it.”  This  was  interpreted  to  mean  that  the 
score  alone  is  not  helpful  but,  with  supporting  data  such  as  the  average  PME  scores, 
average  sub  area  scores  and  average  project  success  ratings,  the  metric  could  have 
more  meaning. 
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3. 


Actionable 


The  metric  was  considered  actionable  by  participants  (M=7,  SD=1.7).  The  low 
variability  in  this  score  also  indicates  a  strong  consensus.  It  was  noted  by  a  subject  that 
the  areas  where  improvement  was  required  was  clear;  however,  it  was  hard  to  prioritize 
which  area  to  target  first.  The  subject  stated,  “Realistically,  I  am  not  going  to  be  able  to 
address  each  of  the  low  scoring  areas  simultaneously,  so  if  I  have  to  pick  an  area  of 
improvement,  I  want  to  pick  the  one  that  is  going  to  give  me  the  best  chance  of 
improving  my  project  success  and  that  may  not  be  the  one  with  the  lowest  score.” 
Another  subject  asserted  that  when  an  area  is  performing  poorly,  by  a  large  gap, 
compared  to  others  it  provides  clear  insight  for  improvement  initiatives  but  in  other 
cases  it  will  be  less  clear  what  action  to  take.  The  metric  does  not  currently  provide 
specific  data  on  questions  in  the  SPMEI  but  one  respondent  provided  an  excellent  idea: 
“In  order  to  begin  self  improvement  it  would  be  good  to  see  a  breakdown  of  key 
techniques  in  each  (sub)  area  and  how  you  scored  on  each.  That  way  you  could  begin 
focusing  of  (specific)  techniques  you  were  lacking  in.” 

4.  Ambiguity 

For  ambiguity,  the  metric  scored  an  average  of  7  (SD=1.8).  It  was  reported,  by 
one  participant,  that  the  scores  did  not  tell  if  they  had  done  well  or  not.  On  a  positive 
note,  the  sub  areas  satisfied  another  respondent,  who  commented  that  they  created 
clear  boundaries  and  that  the  sub  area  descriptions  were  simple  to  understand. 

5.  Reliability 

It  was  pointed  out  by  a  subject  that  the  reliability  of  the  metric  is  inherently  related 
to  the  reliability  of  the  source.  In  other  words,  the  respondent  must  have  a  thorough 
knowledge  of  the  project  management  practices  in  place  for  the  metric  result  to  be 
reliable.  One  of  the  assumptions  of  using  the  tool  is  that  it  should  be  used  by  a  person 
who  has  extensive  knowledge  on  all  areas  of  the  project.  The  reliability  score  had  a 
mean  of  7  with  a  range  of  4. 
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6.  Accuracy 


The  metric  was  considered  to  be  accurate  by  the  subjects  (M=7,  SD=1.6).  One 
respondent  found  the  metric  to  be  very  accurate  and  said  that  it  reflected  the  weak  and 
strong  areas  he  instinctively  felt  the  project  had.  The  accuracy  scores  were  the  most 
consistent  across  all  of  the  responses,  shortly  followed  by  reliability. 

7.  Timely 

As  a  timely  metric,  the  PME  score  was  rated  similarly  to  reliability  (M=7,  S=1 .8). 
It  was  pointed  out  by  a  subject  that  if  the  PME  score  was  produced  after  the  initial 
requirements  phase,  then  it  would  help  the  project  manager  grasp  what  type  of  project 
management  activities  still  need  to  be  carried  out.  This  confirmed  an  original 
assumption  that  the  measurement  activity  should  be  conducted  after  the  initial 
requirements  phase  of  the  project. 

8.  Predictability 

The  metric  was  considered  to  have  weaker  predictive  attributes  by  the  subjects 
(M=6,  SD=2.1).  One  participant  commented  that  some  of  the  sub  area  scores  could  be 
used  in  a  predictive  way,  such  as  the  stakeholder  involvement  score;  however,  other 
sub  areas  were  considered  less  predictive  (i.e.,  teamwork).  Another  participant  stated 
that  they  would  not  use  the  instrument  as  a  predictive  tool. 

Five  out  of  six  participants  said  they  would  use  the  metric  on  the  next  project  they 
worked  on.  Although  not  seen  as  a  particularly  predictive  metric,  the  majority  of 
respondents  found  the  metric  useful.  It  was  generally  seen  to  be  helpful  in  identifying 
strength  and  weaknesses.  The  low  performing  sub-project  management  areas  could  be 
selected  for  improvement  action.  It  was  also  generally  agreed  that  the  measurement 
could  be  used  to  monitor  the  evolution  of  the  software  project  management  practices 
over  time.  On  the  negative  side,  the  questions  in  the  SPMEI  were  considered  open  to 
interpretation  in  certain  areas. 
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Metric  Feedback 
Score 


Responses 


Figure  21 .  Metric  Feedback  Scores  for  Each  Participant 
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Table  29.  Metric  Feedback  Quantitative  Results 


Manageable 

Meaningful 

Actionable 

Ambiguity 

Reliability 

Accuracy 

Timely 

Predictive 

Yes/No 

Score 

Score(%) 

Participant  1 

9 

7 

7 

7 

9 

6 

7 

6 

Yes 

58 

73% 

Participant  2 

9 

7 

7 

6 

7 

6 

8 

7 

No 

57 

71% 

Participant  3 

7 

8 

8 

8 

8 

7 

9 

9 

Yes 

64 

80% 

Participant  4 

4 

7 

6 

3 

5 

7 

5 

4 

Yes 

41 

51% 

Participant  5 

8 

8 

8 

8 

8 

8 

8 

8 

Yes 

64 

80% 

Participant  6 

10 

8 

7 

8 

9 

9 

9 

8 

Yes 

68 

85% 

Participant  7 

4 

4 

3 

8 

5 

4 

4 

3 

No 

35 

44% 

Participant  8 

8 

10 

8 

8 

8 

8 

8 

5 

Yes 

63 

79% 

Min 

4 

4 

3 

3 

5 

4 

4 

3 

35 

0.4375 

Max 

10 

10 

8 

8 

9 

9 

9 

9 

68 

0.85 

Range 

6 

6 

5 

5 

4 

5 

5 

6 

33 

0.4125 

Mean 

7 

7 

7 

7 

7 

7 

7 

6 

75% 

56 

70% 

Std  Dev 

2.26 

1.68 

1.66 

1.77 

1.59 

1.55 

1.83 

2.12 

0.46 

11.90 

0.14 
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VI.  DISCUSSION  AND  CONCLUSION 


With  the  complexity  of  software  projects  increasing  every  year,  project 
managers  need  new  tools  to  tackle  these  new  system  developments.  A  tool  that 
measures  the  effectiveness  of  software  project  management  could  be  used  to 
identify  the  management  strengths  and  weaknesses  and  allow  projects  to  make 
improvements  to  their  practices  in  order  to  increase  their  likelihood  of  success. 
One  tool  that  does  this  is  the  Software  Project  Management  Effectiveness  Metric. 

The  purpose  of  this  study  was  to  measure  the  software  project 
management  effectiveness  of  recent  software  projects,  using  the  software  project 
management  effectiveness  metric,  and  obtain  the  opinions  of  practicing  software 
professionals  on  the  applicability  and  usefulness  of  the  metric. 

Nine  software  projects  were  measured  using  the  software  project 
management  evaluation  instrument  and  a  PME  metric  report  was  produced  for 
each.  A  correlation  analysis  was  conducted  on  the  measured  variables,  PME 
score  and  Project  Success  Rating,  combined  with  those  from  previous  research. 
Six  of  the  projects  in  the  study  reviewed  their  respective  PME  score  and  then 
completed  a  further  survey  that  sought  data  on  the  practicality  and  applicability  of 
the  metric. 

A.  DISCUSSION 

An  important  finding  that  needs  to  be  highlighted  is  the  relationship 
between  the  PME  metric  and  the  average  staff  size  of  a  project.  The  correlation 
of  this  relationship  was  very  low  at  0.29.  This  shows  that  the  metric  does  not 
favor  projects  of  any  particular  size.  This  indicates  that  the  PME  can  be  used  on 
any  project  size.  However,  a  project  manager  should  be  most  comfortable  using 
the  metric  on  projects  with  a  staff  size  of  at  least  four.  This  is  because  a  more 
formal  project  management  approach  is  typically  used  and  required  when  project 
teams  approach  four  or  more.  When  the  project  staff  size  is  below  four  it  is 
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assumed  that  many  project  management  practices  in  the  framework  would  be 
unnecessary  because  the  system  development  complexity  would  be  less.  For 
instance,  a  three-man  web  development  effort  may  be  a  small  business  with  no 
project  manager,  quality  department  or  organizational  hierarchy.  It  is 
recommended  that  the  metric  be  used  on  projects  with  a  staff  size  of  four  (or 
more)  when  a  formal  project  management  approach  is  required  and  in  place. 

Some  noteworthy  results  were  discovered  about  specific  project 
management  areas  and  practices.  Firstly,  the  project  manager  sub  area  had  the 
highest  correlation  (r=  0.69)  with  project  success  out  of  every  single  score.  This 
corroborates  well  with  Verner  and  Evanco’s  pronouncement  that  an  above- 
average  project  manager  was  positively  associated  with  project  success  (Verner 
&  Evanco,  2005).  Secondly,  the  risk  management  main  area  was  positively 
correlated  with  project  success  (r=0.67).  In  a  similar  way,  Verner  and  Evanco 
surmised  that  managing  risks  throughout  the  project  was  significantly  associated 
with  project  success.  But  ironically,  risk  management  was  the  least  practiced 
project  management  discipline  (Verner  &  Evanco,  2005).  This  was  also  found  to 
be  the  case  in  this  research.  The  average  risk  management  score  was  5.6 
(approximately  one  point  below  the  other  main  area  scores).  Projects  found  to  be 
deficient  in  these  areas  should  concentrate  their  improvements  efforts  here. 

The  relationship  between  the  PME  score  and  the  project  success  rating 
was  identified  as  having  a  strong  positive  correlation  (r=0.68).  The  correlation 
found  in  this  study’s  combined  data  set  was  exactly  the  same  as  the  correlation 
calculated  in  Demir’s  study.  It  was  not  expected  to  be  the  exact  same  value  but 
the  r  value  found  in  this  study  was  expected  to  be  above  0.5.  This  study  has 
independently  verified  the  strong  correlation  between  these  two  variables  as 
reported  by  Demir. 

The  SPMEI  itself  was  generally  seen  by  participants  to  have  a  noticeable 

portion  of  ambiguous  questions.  One  subject  reported,  “The  (SPMEI)  questions 

need  to  be  less  open  to  interpretation”  and  another  said,  “Reduce  scope  to 

questions  that  could  be  answered  objectively.”  It  was  suggested  that  some 
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examples  integrated  into  the  questions  would  remove  the  ambiguity.  A  good 
example  of  this  type  of  ambiguity  is  present  in  one  of  the  risk  control  questions, 
when  the  subject  is  asked  if  the  risks  are  managed  as  they  occur.  A  risk  is  a 
future  event  that  may  or  may  not  occur.  If  a  risk  occurs,  it  is  a  problem  impacting 
on  the  projects  objectives.  This  type  of  question  can  be  confusing.  The  SPMEI 
should  be  reviewed  for  ambiguity. 

This  is  the  first  study  where  the  metric  scores  were  provided  to  the 
participants  and  they  were  asked  for  their  feedback  on  the  practicality  of  the 
metric.  It  was  found  that  75%  of  respondents  would  use  the  metric  on  the  next 
project  they  worked  on.  More  research  needs  to  be  completed  in  order  for  the 
tool  to  be  used  a  predictive  measure.  With  more  data,  the  metric  can  be  studied 
to  identify  its  predictive  attributes. 

B.  LIMITATIONS 

The  external  validity  of  the  study  is  a  weakness  due  to  the  small  sample 
size.  It  was  difficult  to  find  participants  to  complete  the  SPMEI  surveys  even  if 
they  indicated  interest  during  initial  communications.  Out  of  all  the  people 
contacted  through  the  networking  approach,  there  was  a  53%  SPMEI  response 
rate.  However,  the  combined  data  set  of  25  projects  now  represents  the  largest 
sample  size  for  the  software  project  management  effectiveness  measurement 
tools  covered  in  the  literature  review. 

Due  to  vast  size  of  the  software  industry,  it  is  fair  to  assume  that  the 
sample  is  not  a  fair  representation  of  the  software  project  population  around  the 
globe.  At  the  same  time,  it  is  not  possible  to  identify  what  a  representative 
sample  would  be,  due  to  the  lack  of  published  data  about  the  software 
development  industry. 

The  correlation  analysis  depends  on  the  accuracy  of  the  PME  score  and 
the  project  success  ratings.  The  project  success  rating  is  a  purely  subjective 
score.  Subjects  were  asked  at  the  start  of  the  SPMEI  to  provide  a  rating,  and 
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again  at  the  end.  In  66%  of  responses,  the  rating  given  at  the  end  of  the  survey 
differed  from  the  rating  given  at  the  start  of  the  survey.  This  is  an  indication  of 
how  subjective  the  rating  is  and  obviously  the  correlation  analysis  is  affected  by 
the  subjective  nature  of  the  success  rating.  If  this  research  was  to  be  conducted 
again,  it  would  be  beneficial  to  have  multiple  opinions  on  the  success  rating  of 
the  project  and  then  the  mean  could  be  used  for  correlation  analysis.  Another 
way  could  be  to  provide  more  objective  criteria  for  project  success  ratings. 

Many  participants  skipped  the  essay-type  questions  posed  in  the  metric 
feedback  instrument.  Additionally,  many  of  the  essay-type  answers  were  difficult 
to  interpret.  If  the  feedback  instrument  was  to  be  used  again,  a  post-survey 
interview  should  be  conducted  to  ask  questions  that  respondents  skipped  and  to 
clarify  their  answers. 

C.  RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

The  SPMEI  sample  size  could  still  benefit  from  substantial  growth.  While 
building  numbers  is  important,  it  is  more  critical  for  future  research  using  the 
metric  to  concentrate  on  unsuccessful  projects  and  projects  with  medium  to  large 
staff  size.  Sampling  these  types  of  projects  will  fill  a  visible  gap  in  the  current 
sample  and  provide  new  insights  to  the  lower  end  of  the  success  spectrum. 

The  SPMEI  was  not  changed  at  all  for  this  study.  As  mentioned 
previously,  the  SPMEI  suffers  from  a  degree  of  ambiguity  in  its  questions.  The 
SPMEI  would  benefit  from  a  revision  of  the  questions  to  decrease  ambiguity. 
Additionally,  the  SPMEM  score  weightings  for  individual  questions  could  be 
revised  based  on  a  correlation  analysis  of  responses  and  the  success  ratings. 
This  research  could  be  conducted  in  a  similar  way  to  Ivan  and  Evanco’s  study 
described  in  Chapter  II. 

The  subjectivity  of  the  SPMEI  has  still  not  been  quantitatively  analyzed. 
This  has  reliability  and  accuracy  implications  for  the  SPME  metric.  To  garner 
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information  on  the  subjectivity  of  the  SPMEI,  a  study  should  be  conducted  where 
at  least  two  personnel  complete  the  SPMEI  and  a  comparison  of  the  results  is 
made. 

The  manageability  of  the  SPMEI  and  SPMEM  was  a  concern.  In  order  to 
make  the  metric  more  manageable,  the  SPMEI  can  be  broken  down  into  its  sub 
areas  or  main  areas  and  distributed  to  different  personnel  on  the  project.  The 
results  can  be  combined  and  a  PME  score  can  then  be  produced.  To  test  the 
applicability  of  this  approach,  one  measurement  can  be  obtained  from  multiple 
participants  and  another  measurement  can  be  made  using  a  separate  single 
participant.  The  two  PME  scores  can  be  compared  for  accuracy.  Splitting  the 
SPMEI  up  into  sub  areas  for  completion  shares  the  burden  of  completing  the 
survey  among  the  project  team  members.  Such  an  approach  may  require  some 
redesign  of  the  SPMEI  and  SPMEM  as  it  was  originally  intended  to  be  completed 
by  one  person  only. 

D.  CONCLUSION 

The  present  study  illuminated  some  salient  findings  within  the  area  of 
software  project  management  effectiveness  measurement.  First,  all  the  projects 
that  scored  a  software  project  management  effectiveness  metric  score  of  6  or 
greater  in  this  study  were  rated  as  a  success.  Out  of  the  22  successful  projects  in 
the  study,  72%  had  a  PME  score  of  6  or  above.  It  was  verified  that  the  PME 
score  had  a  strong  positive  correlation  with  the  project  success  rating.  From 
these  results,  it  can  be  concluded  that  effective  project  management  is  a 
determinant  in  the  success  of  the  software  projects.  If  a  project  has  a  PME  score 
of  six  or  greater,  then  they  are  on  the  right  path  to  improving  their  probability  of 
project  success. 

Second,  it  was  revealed  by  a  correlation  analysis  that  the  metric  can  be 
projects  with  a  wide  range  of  staff  sizes.  Although  it  is  recommended  that 
projects  have  at  least  four  members  before  applying  the  measurement,  it  is  still  a 
great  tool  for  other  relatively  small  projects  who  do  not  wish  to  invest  the  time 
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and  effort  in  getting  a  CMMI  appraisal.  The  metric  can  be  used  as  a  much  more 
lightweight  tool  to  improve  project  management  practices.  On  the  other  hand,  it 
could  also  assist  with  preparing  for  a  CMMI  appraisal  as  well. 

Lastly,  probably  the  most  important  conclusion  is  that  the  currently 
practicing  software  professionals  who  took  part  in  this  study  were  exceedingly 
interested  in  using  the  metric  on  their  next  project.  Seventy-five  percent  of 
respondents  said  they  wanted  to  use  the  metric.  It  can  safely  be  assumed  that 
this  tool  needs  to  be  put  into  practice  immediately  and,  based  on  the  results, 
project  managers  should  be  aiming  to  achieve  a  PME  score  of  at  least  six  as 
soon  as  practical.  The  practitioner  feedback  has  helped  to  further  substantiate 
the  accuracy  and  usefulness  of  the  SPME  metric. 

Software  project  management  is  a  relatively  new  discipline,  having  only 
emerged  in  the  latter  half  of  the  last  century.  A  new  discipline  requires  new  tools. 
Like  any  metric,  the  software  project  management  effectiveness  metric  should 
not  be  the  one  and  only  metric  used  on  a  project.  But  project  managers  should  at 
least  consider  putting  it  in  their  tool  kit.  A  metric  that  measures  the  effectiveness 
of  software  project  management  can  be  used  to  evaluate,  monitor  and  improve 
the  project  management  practices.  This  metric  can  clearly  be  used  to  identify  the 
strengths  and  weaknesses  of  current  project  management  practices  and  produce 
meaningful  quantitative  results.  The  metric  shows  the  most  promise  as  a  post¬ 
mortem  tool.  Post-mortem  reviews  are  important  for  process  improvement,  but 
projects  seldom  perform  them.  As  a  result,  they  tend  to  repeat  the  same 
mistakes  project  after  project.  This  metric  could  be  the  awakening  that  some 
software  project  managers  need,  and  a  gateway  to  more  success. 
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APPENDIX  A.  GLOSSARY 


Communication 

It  is  the  exchange  of  ideas,  opinions  and 
information  through  written  or  spoken  words, 
symbols  or  actions. 

Configuration 

Management 

A  discipline  applying  technical  and  administrative 
direction  and  surveillance  to  (1)  identify  and 
document  the  functional  and  physical 
characteristics  of  a  configuration  item,  (2)  control 
changes  to  those  characteristics,  (3)  record  and 
report  change  processing  and  implementation 
status,  and  (4)  verify  compliance  with  specified 
requirements. 

Leadership 

The  ability  to  lead,  including  inspiring  others  in  a 
shared  vision.  Leaders  have  clear  visions  and 
they  communicate  these  visions  to  their 
employees.  They  foster  an  environment  within 
their  companies  that  encourages  risk  taking, 
recognition  and  rewards,  and  empowerment 
allowing  other  leaders  to  emerge. 

Organizational 

Commitment 

Organizational  commitment  is  the  employee's 
psychological  attachment  to  the  organization  and 
organizational  goals. 

PME  Metric 

Refer  to  Software  Project  Management  Metric 

Process 

A  sequence  of  steps  performed  for  a  given 
purpose;  for  example  the  software  development 
process. 

Project  Monitoring  & 
Control 

Project  monitoring  is  the  process  of  keeping  the 
project  and  project  related  factors  under 
observation.  Project  control  is  to  ensure  that 
project  goes  according  to  what  is  planned  and 
deviations  from  the  plan  kept  under  control. 

Project 

Planning/Estimation 

Project  planning  is  the  process  to  quantify  the 
amount  of  time  and  budget  a  project  will  cost.  The 
purpose  of  project  planning  is  creating  a  project 
plan  that  a  project  manager  can  use  to  track  the 
progress  of  his  team.  Estimation  includes  creating 
estimates  of  project  cost  and  schedule  using 
various  tools  and  techniques. 

Quality  Engineering 

In  engineering,  quality  control  and  quality 
engineering  are  involved  in  developing  systems  to 
ensure  products  or  services  are  designed  and 
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produced  to  meet  or  exceed  customer 
requirements.  It  involves  all  activities  and 
commitment  towards  development  of  a  high 
quality  product  to  meet  or  increase  the 
customer/user  satisfaction. 

Requirements 

Management 

The  management  of  all  requirements  received  by 
or  generated  by  the  project,  including  both 
technical  and  nontechnical  requirements  as  well 
as  those  requirements  levied  on  the  project  by  the 
organization. 

Risk  Assessment 

A  process  or  a  set  of  activities  that  involves 
measurement  of  risks  to  determine  priorities  and 
to  enable  identification  of  appropriate  level  of  risk 
treatment. 

Risk  Control 

That  part  of  risk  management  which  involves  the 
implementation  of  policies,  standards,  procedures 
and  physical  changes  to  eliminate  or  minimize 
adverse  risks. 

Scope  Management 

Scope  management  is  the  process  of  keeping 
track  of  scope  changes  and  limiting  the  changes 
to  the  point  that  they  are  not  disruptive  to  the 
success  of  the  project. 

Software  Project 
Management 

Effectiveness  Metric 

This  metric  is  a  measure  of  the  project 
management  effectiveness  in  a  software  project. 

It  captures  the  effectiveness  of  the  project 
management  from  the  start  of  the  project  to  the 
point  in  time  of  the  measurement. 

Staffing  &  Hiring 

Staffing  is  the  practice  of  finding,  evaluating,  and 
establishing  a  working  relationship  with  future 
colleagues  on  a  project  and  firing  them  when  they 
are  no  longer  needed.  Staffing  involves  finding 
people,  who  may  be  hired  or  already  working  for 
the  company  (organization)  or  may  be  working  for 
competing  companies. 

Stakeholder  Involvement 

Stakeholder  involvement  is  the  early  and 
extensive  engagement  of  stakeholders  in  the 
process  of  planning,  decision  making,  and 
implementation  of  a  project. 

Supplementary  Activities 

Supplementary  activities  are  activities  conducted 
which  are  not  directly  related  to  the  project 
outcome.  However,  these  activities  indirectly 
increase  the  success  probability  of  the  project. 
Such  activities  include  use  of  project 
management,  development,  testing  and  other 
types  of  tools,  training  of  the  personnel,  logistics, 
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increasing  the  satisfaction  of  the  work 
environment  etc. 

Teamwork 

Teamwork  is  the  concept  of  people  working 
together  towards  a  common  goal  set  as  a  team. 

Technical  Complexity 

Technical  complexity  refers  to  the  complexity  of 
the  design,  product,  project  deliverables  and 
technologies  used  in  the  development  of  the 
product. 
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APPENDIX  B.  SPMEI 


Consent  to  Participate  in  Research 

Introduction.  You  are  invited  to  participate  in  a  research  study  entitled  “The 
Effectiveness  of  Software  Project  Management  Practices”  being  conducted  by  the 
Naval  Postgraduate  School. 

Procedures.  The  goal  of  this  study  is  to  gather  information  on  software  project 
management  practices.  You  will  be  asked  to  fill  out  a  questionnaire  which  will  take 
approximately  90  minutes  depending  on  the  participant.  The  questionnaire  is  only 
related  to  the  research  and  serves  no  purpose  other  than  this  research  endeavor. 

Voluntary  Nature  of  the  Study.  Your  participation  in  this  study  is  strictly 
voluntary.  If  you  choose  to  participate  you  can  change  your  mind  at  any  time  and 
withdraw  from  the  study.  You  will  not  be  penalized  in  any  way  or  lose  any  benefits 
to  which  you  would  otherwise  be  entitled  if  you  choose  not  to  participate  in  this 
study  or  to  withdraw. 

Potential  Risks  and  Discomforts.  The  potential  risks  of  participating  in  this 
study  are: 

a)  A  breach  of  confidentiality  may  result  in  embarrassment  of  the  research 
subject. 

Anticipated  Benefits.  Anticipated  benefits  from  this  study  are: 

a)  To  assist  in  the  development  of  project  management  metrics  and  improve  the 
software  engineering  body  of  knowledge  to  improve  software  project 
management;  and 

b)  To  enable  the  development  of  a  tool  for  you  to  monitor,  evaluate  and  improve 
your  projects. 

Compensation  for  Participation.  No  tangible  compensation  will  be  given.  A 
copy  of  the  research  results  will  be  available  at  the  conclusion  of  the  experiment. 

Confidentiality  &  Privacy  Act.  Any  information  that  is  obtained  during  this  study 
will  be  kept  confidential  to  the  full  extent  permitted  by  law.  All  efforts,  within  reason, 
will  be  made  to  keep  your  personal  information  in  your  research  record  confidential 
but  total  confidentiality  cannot  be  guaranteed.  No  information  will  be  publicly 
accessible  which  could  identify  you  as  a  participant.  Research  records  will  be 
stored  and  maintained  in  electronic  form  on  NPS  secure  servers  only  accessible  by 


103 


the  researchers.  Any  hard  copy  material  containing  research  findings,  including  a 
thesis,  will  not  contain  any  personal  information. 

Points  of  Contact.  If  you  have  any  questions  or  comments  about  the  research,  or 
you  experience  an  injury  or  have  questions  about  any  discomforts  that  you 
experience  while  taking  part  in  this  study  please  contact  the  Principal  Investigator, 
Dr  John  Osmundson,  831-656-3775,  iosmundson@nps.edu.  Questions  about  your 
rights  as  a  research  subject  or  any  other  concerns  may  be  addressed  to  the  Naval 
Postgraduate  School  IRB  Chair,  CAPT  John  Schmidt,  USN,  831-656-3864, 
jkschmid@nps.edu. 

Statement  of  Consent.  I  have  read  the  information  provided  above.  I  have  been 
given  the  opportunity  to  ask  questions  and  all  the  questions  have  been  answered 
to  my  satisfaction.  I  have  been  provided  a  copy  of  this  form  for  my  records  and  I 
agree  to  participate  in  this  study.  I  understand  that  by  agreeing  to  participate  in 
this  research  and  signing  this  form,  I  do  not  waive  any  of  my  legal  rights. 

Dear  Fellow  Colleagues, 

I  sincerely  appreciate  you  taking  time  to  participate  in  this  study.  This  study  is 
conducted  as  part  of  my  postgraduate  thesis  research  at  the  Naval  Postgraduate 
School.  My  colleagues  and  I  are  testing  the  applicability  of  a  software  project 
management  self-evaluation  instrument  (put  simply,  a  questionnaire).  We  would 
like  you  to  apply  the  instrument  on  a  software  project  you  have  worked  on.  Your 
participation  will  be  completely  anonymous. 

How  we  plan  to  use  your  responses 

The  anticipated  benefits  of  this  study  are: 

a)  to  assist  in  the  development  of  project  management  metrics;  and 

b)  to  identify  practices  which  increase  the  chances  of  project  success;  and 

c)  to  assist  in  the  development  of  a  tool  for  managers  to  monitor,  evaluate  and 
improve  their  projects. 

The  only  requirements  for  your  participation  are  the  following 

a)  you  have  worked  on  a  software  intensive  development  project  in  the  past;  or 

b)  you  are  currently  working  on  a  software  intensive  development  project  that  has 
completed  the  initial  requirements/inception/conceptual  phase;  and 

c)  you  have  a  broad  knowledge  of  the  project  management  practices  in  place  on  your 
project. 
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What  personal  information  will  be  collected: 

The  questionnaire  investigates  what  happened  during  a  particular  project 
development.  This  is  NOT  an  evaluation  of  the  project  manager,  the 
management  team,  or  any  other  person.  This  instrument  is  not  designed  for  that 
purpose.  Any  inference  derived  for  such  a  purpose  will  definitely  be  incorrect  and 
misleading.  This  is  NOT  an  evaluation  of  the  organization.  It  focuses  on  the 
project  only. 

How  your  response  will  be  handled 

This  study  will  be  conducted  with  discretion  and  the  highest  regard  for  your 
confidentiality.  In  the  final  published  research  results  it  will  not  be  possible  to 
trace  the  results  back  to  a  particular  person,  organization,  or  any  entity.  Your 
response  will  only  be  identified  as  an  identification  code  on  all  data  collection 
forms. 


Your  identification  code  is:  XXX 

Please  find  the  questionnaire  attached.  If  you  have  questions  about  the  study  or 
the  research,  please  do  not  hesitate  to  contact  me. 

Yours  Sincerely, 


Christopher  Cullen 

Flight  Lieutenant 
Computer  Science  Department 
Naval  Postgraduate  School 
Monterey  CA  93943 

Tel:  1-831-917-5255 
Fax:  1-831-333-9277 

Email:  ccullen@nps.edu 
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DIRECTIONS  FOR  FILLING  OUT  THE  QUESTIONNAIRE 

•S  There  are  16  sections  in  the  questionnaire.  It  takes  approximately  90  minutes  to 
complete,  depending  on  the  participant.  The  questionnaire  examines  from  the  start  of 
the  project  until  it  is  delivered  to  the  customer  for  the  first  time  (or  it  is  cancelled). 

■S  Choose  a  project  you  have  worked  on  and  have  extensive  knowledge.  The  project  you 
choose  does  not  have  to  be  a  complete  success  -  it  may  have  had  moderate  success, 
poor  success  or  could  even  have  been  cancelled.  We  are  interested  in  analysing  the 
entire  spectrum  of  software  projects. 

■f  You  may  respond  to  the  questionnaire  sections  in  any  order  you  like  and  you  do  not 
have  to  complete  the  survey  in  one  sitting. 

•S  The  questions  are  straightforward  and  designed  to  be  simple  and  easy  to  understand. 
There  are  two  main  types  of  questions.  In  the  first  type,  simply  check  one  or  more 
statements  that  apply  to  the  project. 

Check  the  STATEMENT  that  applies  to  the  project.  (CHECK  ONLY  ONE) 

□  x 

M  Y 

I~1  None 

Check  the  STATEMENT/S  that  applies  to  the  project.  (CHECK  ALL  THAT  APPLY) 

Sx 

□  Y 

0Z 

I~1  None 

■S  In  the  second  type,  simply  check  whether  you  agree  or  not  on  a  particular  statement. 


Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

SI1 

STATEMENT 

Agree 

Disagree 

Applicable 

□ 

□ 

□ 

□ 

□ 

•S  When  there  are  combined  statements,  consider  them  as  one  concept  and  respond  as  is, 
or  take  an  average  of  the  ratings  for  each  of  the  statement. 

•S  The  questionnaire  is  designed  as  a  whole.  Trying  to  infer  results  from  just  one  or 
more  sections  will  be  misleading. 

•S  Please  respond  to  all  questions.  Thanks  again  for  your  participation! 
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GENERAL  PROJECT-RELATED  QUESTIONS  (17  Questions  -  About  5  minutes) 
Directions:  Please  provide  responses  to  the  following  questions  to  the  best  of  your  knowledge. 

ENTER  THE  CODE  PROVIDED: 


PR1. 

What  was  the  goal  of  the  project?  What  kind  of  an  application  was  developed?  What  were  the  deliverables?  Please 
briefly  state. 

PR2. 

What  was  the  title  of  the  project  (if  there  is  one)? 

PR3. 

What  was  the  projected/planned  effort  for  the  project?  (in  terms  of 
man-month) 

Man-month 

PR4. 

What  was  the  actual  effort  for  the  project?  (in  terms  of  man-month) 

Man-month 

PR5. 

What  was  the  actual  cost  of  the  project? 

Dollars 

PR6. 

What  was  the  projected/planned  budget  for  the  project? 

Dollars 

PR7. 

How  long  did  the  project  take? 

*From  start  (or  contract)  date  to  delivery  date 

Months 

PR8. 

What  was  the  projected/planned  schedule  for  the  project? 

Months 

PR9. 

What  was  the  start  date  of  the  project?  (Month/Year) 

/ 
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PR10. 

What  was  the  delivery  date  of  the  project?  (Month/Year) 

/ 

PR11. 

How  much  of  the  functionality  (or  number  of  features)  are  delivered 
to  the  customer?  (Between  the  initial  baseline  and  the  delivered 
product) 

% 

PR12. 

How  many  people  worked  on  the  project?  (Including  the  management,  consultants/contractors,  etc.) 

Requirements  Phase 

Design  Phase 

Implementation  Phase 

Testing  and  Delivery  Phase 

Total 

Or 

Average  number  of  people  from  start  to  end 

PR13. 

What  is  the  size  of  the  project?  (in  terms  of  Lines  of  Code  (KLOC)  or  KLOC 

function  points  (FP) )  FP 

PR14. 

Where  was  the  project  developed?  Which  state,  country,  or  countries? 

PR15. 

What  kind  of  an  organization  developed  the  project?  (government,  commercial,  open  source  community, 
government  contract,  etc.)  Organization  name? 

PR16. 

What  was  the  CMMI  level  of  the  organization  when  the  project  was  undertaken?  0-  5 
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PR17. 

How  woul 
success.) 

d  you  rate  the  overall  success  of  the  project?  (0  beinc 

j  complete  failure  and  10  being  the  complete 

9  10 

]  □ 

012345678 

□  □□□□□□□□□ 

PR18. 

What  is/was  your  role  in  the  project? 

INDEX  (You  can  click  to  jump  to  a  section) 


Communication 

Teamwork 

Leadership 

Organizational  Commitment 

Project  Manager 
Reguirement  Management 

Stakeholder  Involvement 
Project  Monitoring  and  Control 

Project  Planning  and  Estimation 

Scope  Management 

Risk  Control 
Staffing  and  Hiring 
Configuration  Management 

Risk  Assessment 
Quality  Engineering 
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COMMUNICATION  Section  (23  Questions  -  About  7-12  minutes) 

Cl.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  A  common  glossary/terminology  for  the  project  is  created. 

□  Communication  procedures  adapts  due  to  changing  project  environment. 

□  Communication  procedures  are  always  followed  as  stated  in  the  communication  planning  documentation  (or  similar  document). 

□  There  is  a  project  information  distribution  list  (or  a  similar  document)  and  it  is  maintained. 

□  The  project  budget  includes  resources  for  communication  and  project  information  distribution  efforts. 

□  None 

C2.  Who  are  generally  present  in  the  project  status  meetings?  (Check  all  that  apply.) 

□  Project  manager 

□  Project  team  leaders 

□  Project  team  members 

Q  Customer/s  and/or  user  representatives 
Q  Various  stakeholders  or  stakeholder  representatives 

□  Executive  management  /  Project  sponsor 


C3.  Which  of  the  following/s  is/are  discussion  items  in  project  status  meetings?  (Check  all  that  apply.) 

□  Project  schedule 

□  Project  budget 

□  Project  risks 

□  Project  staff  problems 

□  Important  development  events  and/or  accomplished  project  deliverables 

□  Requirements 

□  None 

C4.  Which  of  the  following/s  does  the  project  information  distribution  plan/list  (or  similar  document)  contain?  (Check  all  that  apply.) 

□  Project  information  type/context  (What  will  be  communicated) 

□  Recipients  of  various  communication  items  (Stakeholders-  who  should  receive  the  information) 

□  Project  related  information  distribution  frequency 

□  Timeframe  of  the  relevant  communication 

□  Communication  format  and  medium  (How  the  communication  will  be  conducted-  reports,  meetings,  teleconferencing  etc.) 
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□  Responsible  project  staff  for  communication 

□  Not  available 


(5) 

(4) 

(3) 

(2) 

(i) 

N/A 

C5 

The  importance  of  communication  is  understood  and  established  between  stakeholders  and 
project  team  members.  There  is  commitment  to  good  communication. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

m 

Not 

Applicable 

□ 

C6 

Stakeholders  including  project  team  members’  needs  for  various  project  data  and 
information  are  analyzed  and  identified. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

m 

Not 

Applicable 

□ 

C7 

There  have  been  communication  problems  due  to  various  reasons. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

m 

Not 

Applicable 

□ 

C8 

Communication  is  used  as  a  means  to  resolve  conflicts. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

m 

Not 

Applicable 

□ 

C9 

There  are  designated  project  team  members  and  representatives  of  stakeholders  responsible 
for  conducting  communication. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

m 

Not 

Applicable 

□ 

CIO 

Communication  procedures  are  documented  and  distributed  to  stakeholders  and  project 
team  members. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

m 

Not 

Applicable 

□ 

C11 

Communication  and  coordination  for  activities  are  planned  in  the  project  plan. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

m 

Not 

Applicable 

□ 

C12 

The  response  and  acknowledgement  procedures  are  planned  and  documented  in 
the  communication  procedures. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

m 

Not 

Applicable 

□ 

C13 

The  information  needs  of  stakeholders  and  project  team  members  are  satisfied  in 
a  timely  manner  through  appropriate  use  of  communications  media. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

m 

Not 

Applicable 

□ 

C14 

As  a  project  manager  or  a  project  team  member,  1  can  easily  communicate  my 
messages  and  1  can  be  understood. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

m 

Not 

Applicable 

□ 

C15 

A  communications  and  project  information/data  management  system  with 
essential  capabilities  are  in  place.  (Such  as  databases,  mail  servers,  or 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

m 

Not 

Applicable 

□ 

C16 

The  project  environment  facilitates  horizontal  communication  that  is  between 
peers. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

m 

Not 

Applicable 

□ 

C17 

The  project  team  operates  in  a  virtual  environment  rather  than  on  a  face-to-face 
basis. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

_ n _ 

Not 

Applicable 

□ 
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C18 

The  project  status  is  visible  to  every  stakeholder  and  project  team  member. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

rn 

Not 

Applicable 

□ 

C19 

The  project  manager,  management  team,  and  team  leaders  are  always  accessible 
to  project  team  members  in  a  timely  manner. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

rn 

Not 

Applicable 

□ 

C20 

When  1  report  a  project  problem,  1  get  timely  acknowledgement  that  my  message 
has  been  received  and  understood. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

rn 

Not 

Applicable 

□ 

C21 

Informal  communications  within  the  team  and  stakeholders  are  also  an  important 
part  of  project  development  environment. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

rn 

Not 

Applicable 

□ 

C22 

The  project  environment  facilitates  free-format  meetings  for  various  purposes. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

rn 

Not 

Applicable 

□ 

C23 

The  project  environment  facilitates  freedom  in  reporting  of  project  problems. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completel 

y 

Disagree 

_ n _ 

Not 

Applicable 

□ 

TEAMWORK  Section  (30  Questions -About  10  minutes) 

T1.  Which  of  the  following/s  are  clearly  documented  in  the  project  plan  for  each  team  member?  (Check  all  that  apply.) 

□  Responsibility  of  the  team  member 

□  Accountability  of  the  team  member 

□  Authority  of  the  project  manager  and  team  members 

□  Reporting  structure 

□  Interfaces  and/or  communication  channels 

□  None 

T2.  How  many  project  team  members  stayed  with  the  project  until  the  end  according  to  the  project  staffing  plan?  (Check  only  one.) 

□  All  Y  □  Most  □  Some  □  None 


T3.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  Notable  project  accomplishments/milestones/deliverables  are  celebrated  with  social  events  or  parties. 
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p  There  are  problem-solving  meetings  with  the  attendance  of  relevant  project  team  members  and  stakeholders. 

P  Organizational  culture  encourages  problem  solving  sessions  with  the  attendance  of  project  members. 

□  When  a  project  team  member  left  the  team  or  the  member  is  removed,  the  rest  of  the  team  has  understood  the  reasoning. 

□  None 

T4.  Which  of  the  following  activities  are  carried  out  throughout  the  project?  (Check  all  that  apply.) 

□  Social  events/parties 

□  Team  building  training 

□  Introduction  meetings  and  parties 

□  Reward  and  other  types  of  ceremonies 

□  Brainstorming  and  problem  solving  meetings  and  sessions 
[H  Meetings  for  self-assessment  of  team  performance 

□  None 


(5) 

(4) 

(3) 

(2) 

(i) 

N/A 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T5 

The  project  is  adequately  staffed  during  the  project  development. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

T6 

The  organization  structure  and  responsibility/task  matrix  are  clearly  documented 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

and  provided  to  project  team  members. 

□ 

□ 

□ 

□ 

□ 

□ 

T7 

There  are  regular  status  meetings  to  self-assess  the  project  team’s  performance  and 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

morale. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T8 

There  is  an  accepted  shared  vision  for  the  project  within  team  members. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T9 

Team  members  are  involved  in  the  project  planning  effort. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

T10 

Team  members  are  involved  in  decision-making  process  during  project 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

development. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

Til 

The  project  status  is  visible  to  team  members. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

T12 

In  order  to  do  the  work  effectively,  all  necessary  project  data  and  information  is 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

easily  accessible  to  project  members. 

□ 

□ 

□ 

□ 

□ 

□ 

113 


T13 

Training  opportunities  are  created  and  made  available  upon  need  or  at  the  request 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

of  team  members. 

□ 

□ 

□ 

□ 

□ 

□ 

T14 

There  are  more  experienced  project  team  members  than  inexperienced  team 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

members. 

□ 

□ 

□ 

□ 

□ 

□ 

T15 

The  project  environment  facilitates  teaming  up  inexperienced  team  members  with 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

the  experienced  team  members. 

□ 

□ 

□ 

□ 

□ 

□ 

T16 

Rewards  for  achievements  are  handed  out  justifiably  and  made  the  project  team 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

happy. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T17 

There  is  trust  and  respect  among  team  members. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T18 

The  project  team  is  empowered  with  adequate  resources  to  do  their  tasks. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

T19 

The  support  from  upper  management  or  project  sponsor  is  visible  to  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

project  team. 

□ 

□ 

□ 

□ 

□ 

□ 

T20 

The  project  offers  stimulating  and  challenging  work  to  project  team 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

members. 

□ 

□ 

□ 

□ 

□ 

□ 

T21 

The  project  environment  offers  professional  growth  potential  for  team 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

members. 

□ 

□ 

□ 

□ 

□ 

□ 

T22 

The  project  suffers  from  not  having  enough  experienced  or  qualified  team 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

members. 

□ 

□ 

□ 

□ 

□ 

□ 

T23 

Team  members  are  tasked  based  on  their  skills,  capabilities,  ambitions  and 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

interests. 

□ 

□ 

□ 

□ 

□ 

□ 

T24 

The  team  members  are  clear  about  how  their  job  performance  will  be 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

evaluated. 

□ 

□ 

□ 

□ 

□ 

□ 

T25 

The  project  team  members  believe  that  they  have  enough  resources  to 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

accomplish  their  jobs  successfully. 

□ 

□ 

□ 

□ 

□ 

□ 

T26 

The  orientation  procedures  and  the  sponsors  are  documented  and  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

procedures  are  followed  for  the  team  members  joining  the  team  later. 

□ 

□ 

□ 

□ 

□ 

□ 
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T27 

Project  priorities  are  always  made  clear  via  meetings,  presentations  and 
memos;  priorities  are  not  constantly  changing. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

T28 

The  project  suffers  from  lack  of  communication  and  coordination. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

T29 

The  project  suffers  from  lack  of  leadership  at  various  levels. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

T30 

The  project  team  consists  of  people  who  has  worked  together  before. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

LEADERSHIP  Section  (17  Questions  -  About  3-6  minutes) 


(5) 

(4) 

(3) 

(2) 

(i) 

N/A 

Ll 

The  leaders  at  various  levels  promote  competition  rather  than  coordination  within 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

the  project  organization. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

L2 

The  leaders  at  various  levels  sets  example  for  others. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

L3 

After  the  creation  of  the  shared  vision  for  the  project,  the  leaders  at  various  levels 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

maintain  the  vision. 

□ 

□ 

□ 

□ 

□ 

□ 

L4 

The  leaders  at  various  levels  are  effective  problem-solvers  in  technical  and  social 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

issues. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

L5 

The  management  protects  the  team  from  outside  interference. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

L6 

The  leaders  at  various  levels  clearly  state  their  leadership  styles  upfront  with 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

reasons  for  the  style. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

L7 

The  leaders  at  various  levels  assign  correct  tasks  to  correct  people. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

L8 

The  leaders  at  various  levels  are  respected  by  the  team  members. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

L9 

The  leaders  at  various  levels  easily  delegates  authority  when  necessary. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 
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L10 

The  leaders  at  various  levels  observe  the  morale  of  the  staff  and  takes  proactive 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

action  to  boost  the  morale. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

Lll 

The  project  team  suffers  from  coordination  problems. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

L12 

The  project  team  suffers  from  communication  problems. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

L13 

The  leaders  at  various  levels  welcome  communication  of  project  problems  at  any 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

time. 

□ 

□ 

□ 

□ 

□ 

□ 

L14 

The  leaders  at  various  levels  clearly  define  what  is  expected  from  project  team 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

members. 

□ 

□ 

□ 

□ 

□ 

□ 

L15 

The  project  team  members  freely  share  their  desires,  wishes,  and  concerns  with 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

their  leaders. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

L16 

The  leaders  at  various  handle  project  politics  well. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

*  Provide  response  to  either  LI  7  or  LI  8. 

LI  7.  (Answer  only  if  the  project  team  mostly  consists  of  inexperienced  staff)  Check  the  statement  that  applies  to  the  project.  (Check 
only  one.) 

□  The  leaders  at  various  levels  have  to  make  most  decisions  and  direct  the  staff. 

□  The  leaders  at  various  levels  make  most  decisions  with  the  consultation  of  team  members  and  coach  the  staff. 

□  The  leaders  at  various  levels  and  the  team  members  make  decisions  together. 

□  The  leaders  at  various  levels  mostly  oversee  the  decisions  made  by  the  staff  and  delegate  the  tasks. 

LI 8.  (Answer  only  if  the  project  team  mostly  consists  of  experienced  staff)  Check  the  statement  that  applies  to  the  project.  (Check  only 
one.) 

□  The  leaders  at  various  levels  have  to  make  most  decisions  and  direct  the  staff. 

□  The  leaders  at  various  levels  make  most  decisions  with  the  consultation  of  team  members  and  coach  the  staff. 

Q  The  leaders  at  various  levels  and  the  team  members  make  decisions  together. 

□  The  leaders  at  various  levels  mostly  oversee  the  decisions  made  by  the  staff  and  delegate  the  tasks. 
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ORGANIZATIONAL  COMMITMENT  Section  (27  Questions  -  About  7-1 2  minutes) 


(5) 

(4) 

(3) 

(2) 

(i) 

N/A 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OCl 

The  executive  management  is  committed  to  providing  necessary  financial  support. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

OC2 

The  executive  management  is  committed  to  providing  necessary  flexibility  on  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

project  schedule. 

□ 

□ 

□ 

□ 

□ 

□ 

OC3 

The  executive  management  is  committed  to  providing  necessary  flexibility  on  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

project  functionality  and  quality. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC4 

The  executive  management  and  project  organization  is  open  to  change/adaptation. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

OC5 

There  is  encouragement  for  organizational  and  personal  certifications  such  as 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

CMMI,  PMI,  PMP,  ISO  etc. 

□ 

□ 

□ 

□ 

□ 

□ 

OC6 

There  is  commitment  to  quality  by  executive  management,  team  members  and 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

other  stakeholders. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC7 

Adequate  resources  are  set  aside  for  the  success  of  the  project. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

OC8 

There  is  support  for  bringing  in  expertise  when  needed  (Such  as  technical,  legal, 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

contracting  etc.) 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC9 

There  is  support  for  quality  subcontracting  when  needed. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

OCIO 

The  executive  management  supports  /  empowers  /  enables  the  project  manager  to 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

do  his  job. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC11 

There  is  continuous  and  observable  support  from  executive  management. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC12 

Leaders  at  various  levels  are  committed  to  the  success  of  the  project. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC13 

Leaders  at  various  levels  are  committed  to  their  team  members. 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

□ 

□ 

□ 
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OC14 

The  project  manager  and  leaders  at  various  levels  are  committed  to  providing 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

continuous  support  in  enabling  the  team  members  to  do  their  work. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC15 

The  project  team  members  are  committed  to  the  accomplishment  of  the  project. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

OC16 

The  project  team  members  show  their  commitment  to  staying  with  the  project  until 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

the  end. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC17 

The  project  team  members  put  extra  effort  for  the  success  of  the  project. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

OC18 

The  project  team  members  lack  motivation  due  to  various  reasons  including 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

external  factors. 

□ 

□ 

□ 

□ 

□ 

□ 

OC19 

The  project  manager  and  the  team  members  don’t  consider  the  project  as  a 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

pleasant  challenge. 

□ 

□ 

□ 

□ 

□ 

□ 

OC20 

The  project  manager  and  the  team  members  consider  the  project  as  a  valuable 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

learning  experience. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC21 

There  is  a  friendly-work  environment. 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC22 

The  project  team  members  publicly  and  explicitly  indicate  their  job  satisfaction. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

OC23 

There  is  commitment  from  various  stakeholders  including  project  team  members, 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

customer,  marketing  and  sales  department(if  applicable)  etc. 

□ 

□ 

□ 

□ 

□ 

□ 

OC24 

Executive  management,  project  manager  and  project  team  members  are  committed 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

to  establishing  effective  project  management  and  control  mechanisms. 

□ 

□ 

□ 

□ 

□ 

□ 

OC25.  Which  of  the  following  item/s  does  the  executive  management  show  commitment  to  providing  support?  (Check  all  that  apply.) 

□  Human  resources 

□  Training  needs 

□  Supplementary  needs  such  as  office  space,  tools,  computer  systems  etc. 

□  None 
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0C26.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

p  The  executive  management  clearly  defines  the  authority  and  responsibility  of  the  project  manager. 

□  The  executive  management  allows  for  realistic  budget  and  schedule. 

□  Training  is  made  available  to  all  team  members. 

□  There  are  some  resignations  in  the  project  organization. 

□  The  project  organization  allows  for  career  development. 

□  None 


PROJECT  MANAGER  Section  (27  Questions  -  About  5-9  minutes) 

PM1.  How  many  project  managers  have  changed  during  the  project  (Turnover)?  (Check  only  one.) 

□  None  □  1  □  2  □  3  or  more 

PM2.  How  many  years  of  experience  does  the  project  manager  have?  (Check  only  one.) 

□  Less  Than  5  QS-IO  □  10-15  □  15-20  □  More  Than  20 

PM3.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  The  project  manager  has  certification  related  to  project  management  such  as  PMP  etc. 

□  The  project  manager  has  worked  on  similar  projects. 

□  The  project  manager  has  worked  as  a  project  manager  before. 

□  The  project  manager  has  worked  as  a  practitioner/developer  before,  therefore  has  technical  background. 

Q  The  project  manager  has  worked  on  different  types  of  projects. 

□  None 

PM4.  Which  of  the  following/s  the  project  manager  has  control  over?  (Check  all  that  apply.) 

□  Budget  □  Schedule  □  Product  Quality  □  Process  Quality  □  Hiring  and  letting  go  □  None 


(5)  (4)  (3)  (2)  (1)  N/A 


PM5 

The  project  manager’s  role,  accountability,  and  responsibilities  are  clearly  defined 
and  communicated  to  stakeholders  including  project  team  members. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM6 

The  project  manager  was  given  adequate  authority  and  control  over  the  project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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PM7 

The  project  manager  has  adequate  project  management  education,  training  and 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

experience. 

□ 

□ 

□ 

□ 

□ 

□ 

As  a  project  manager,  I  have  goals  and  a  clear  vision  related  to  the  project.  /As  a 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

PM8 

team  member,  I  observe  that  the  project  manager  has  goals  and  a  clear  vision 
related  to  the  project. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

As  a  project  manager,  I  am  able  to  maintain  the  continuity  of  the  project  vision.  / 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

PM9 

As  a  team  member,  I  observe  that  the  project  manager  is  able  to  maintain  the 
continuity  of  the  project  vision. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PM10 

As  a  project  manager,  I  am  deeply  committed  to  the  project./As  a  team  member,  I 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

observe  the  deep  commitment  in  the  project  manager. 

□ 

□ 

□ 

□ 

□ 

□ 

As  a  project  manager,  I  am  communicative  and  always  accessible  to  team./As  a 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

PM11 

team  member,  I  observe  that  the  project  manager  is  communicative  and  always 
accessible  to  the  team. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PM12 

As  a  project  manager,  I  motivate  staff  and  other  people  well./As  a  team  member,  I 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

observe  that  the  project  manager  motivates  the  staff  and  other  people  well. 

□ 

□ 

□ 

□ 

□ 

□ 

PM13 

As  a  project  manager,  I  am  a  good  planner  and  organizer./ As  a  team  member,  I 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

observe  that  the  project  manager  is  a  good  planner  and  organizer. 

□ 

□ 

□ 

□ 

□ 

□ 

PM14 

As  a  project  manager,  I  am  an  effective  problem  solver. /As  a  team  member,  I 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

observe  that  the  project  manager  is  an  effective  problem  solver. 

□ 

□ 

□ 

□ 

□ 

□ 

As  a  project  manager,  I  consult  to  and  get  advice  from  stakeholders  and  project 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

PM15 

team  members.  / 1  observe  that  the  project  manager  consults  to  and  gets  advice 
from  stakeholders  and  project  team  members. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PM16 

As  a  project  manager,  I  delegate  easily  when  necessary  ./As  a  team  member,  I 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

observe  that  the  project  manager  delegates  easily  when  necessary. 

□ 

□ 

□ 

□ 

□ 

□ 

As  a  project  manager,  I  use  rewarding  and  punishment  mechanisms  effectively.  /As 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

PM17 

a  team  member,  I  observe  that  the  project  manager  uses  rewarding  and 
punishment  mechanisms  effectively. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 
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PM18 

As  a  project  manager,  I  am  a  people  person./As  a  team  member,  I  observe  that  the 
project  manager  is  a  people  person. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM19 

As  a  project  manager,  I  am  an  effective  team  builder  and  player. /As  a  team 
member,  I  observe  that  the  project  manager  is  an  effective  team  builder  and  player. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM20 

As  a  project  manager,  I  support  my  team  members  in  various  aspects./ As  a  team 
member,  I  observe  that  the  project  manager  supports  the  team  members  in  various 
aspects. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM21 

As  a  project  manager,  I  monitor  every  aspect  of  the  project./As  a  team  member,  I 
observe  that  the  project  manager  monitors  every  aspect  of  the  project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM22 

As  a  project  manager,  I  inform  the  stakeholders  and  my  team  members  well./ As  a 
team  member,  I  observe  that  the  project  manager  informs  the  stakeholders  and  the 
team  members  well. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM23 

As  a  project  manager,  I  clarify  when  the  stakeholders  and  the  team  members  are 
confused  about  an  aspect  of  the  project./As  a  team  member,  I  observe  that  the 
project  manager  clarifies  when  the  stakeholders  and  the  team  members  are 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM24 

As  a  project  manager,  I  am  able  to  see  the  project  as  a  whole./ As  a  team  member,  I 
observe  that  the  project  manager  sees  the  project  as  a  whole. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM25 

As  a  project  manager,  I  understand  the  domain  of  the  project./As  a  team  member,  I 
observe  that  the  project  manager  understands  the  domain  of  the  project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM26 

As  a  project  manager,  I  protect  my  team  members  so  that  their  work  don't  get 
disrupted./As  a  team  member,  I  observe  that  the  project  manager  protects  us  so 
that  our  work  don't  get  disrupted. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM27 

As  a  project  manager,  I  understand  and  foresee  the  project  risks./As  a  team 
member,  I  observe  that  the  project  manager  understands  and  foresees  the  project 
risks. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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REQUIREMENTS  MANAGEMENT  Section  (27  Questions  -  About  5-9  minutes) 

RM1 .  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  There  is  a  requirements  development  document  (how  they  are  gathered  and  developed). 

□  There  is  a  requirements  management  document  (how  they  are  handled). 

□  There  is  an  agreed/negotiated  requirements  baseline. 

□  There  is  a  requirements  baseline  document  and  it  is  managed. 

□  None 

RM2.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  Oral  requirements  are  used. 

□  Written  requirements  are  used. 

□  Requirements  are  formal  -  a  standard  guides  the  development;  have  identifiers  and  traceability  matrix  etc. 

□  Requirements  are  informal  -  requirements  are  just  identified  and  listed. 

□  None 

RM3.  Which  of  the  following  activities  are  conducted  in  the  project?  (Check  all  that  apply.) 

□  Market  surveys  □  Customer/User  interviews  Q  Prototyping  Q  Scenarios/  use  cases  □  Observation  of  the  user  in 

operation  □  None 

RM4.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  Stakeholders  are  identified  prior  to  requirements  development  activities. 

□  Requirements  related  documents  have  versions. 

□  There  is  a  requirements  traceability  matrix  (or  a  similar  document  to  trace  the  requirements  during  all  the  development  activities). 

□  Requirements  volatility  (number  of  requirements  change/  percent  of  number  of  requirements  change  etc.)  metrics  are  collected  and  used. 

□  Testing  team  is  involved  in  the  requirement  development  activities. 

□  None 


(5) 

(4) 

(3) 

(2) 

(i) 

N/A 

RM5 

Requirements  prioritization  is  conducted  and  used  for  development  decisions. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RM6 

All  stakeholders  are  involved  in  the  requirements  development. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RM7 

Users  or  user  representatives  are  involved  in  the  requirements  development. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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RM8 

Stakeholders  show  commitment  to  requirements  stability  during  the  project 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

development. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RM9 

Automated  requirements  development  and  management  tools  are  used. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RM10 

All  requirements  are  traceable. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

RM11 

Product  components  and  project  deliverables  can  be  mapped  to  specific 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

requirements. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RM12 

Requirements  are  clear  /  unambiguous. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RM13 

Requirements  are  complete. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RM14 

There  are  no  inconsistencies  among  requirements. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

RM15 

During  the  project  development,  requirements  related  issues  are  resolved  with  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

negotiation  with  the  customers. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RM16 

Requirements  are  validated  with  the  user,  customer  and  necessary  stakeholders. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

RM17 

There  are  designated  points  of  contact  (people)  representing  various  stakeholders 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

to  resolve  requirements  related  issues. 

□ 

□ 

□ 

□ 

□ 

□ 

RM18 

The  procedures  are  formal  for  requirements  validation  (what  the  customer 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

want). 

□ 

□ 

□ 

□ 

□ 

□ 

RM19 

The  procedures  are  formal  for  requirements  verification  (the  system  does 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

what  requirements  state). 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RM20 

There  is  a  formal  requirements  change  procedure  and  document. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

RM21 

Requirements  history  and  rationale  for  requirements  changes  are 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

recorded. 

□ 

□ 

□ 

□ 

□ 

□ 
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RM22 

Requirements  are  worded  simple  and  each  requirement  consists  of  only 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

one  concept. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RM23 

Extra  effort  is  spent  to  make  the  requirements  testable. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

RM24 

There  are  testing  plans  to  check  if  the  requirements  are  implemented  as 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

intended. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RM25 

User/customer  profiles  are  identified  and  documented. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

RM26 

Requirements  are  constantly  changing  and  all  changes  are  being 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

implemented. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RM27 

Requirements  are  kept  stable  at  some  point. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

STAKEHOLDER  INVOLVEMENT  Section  (12-16  Questions  -  About  3-7  minutes) 


(5) 

(4) 

(3) 

(2) 

(i) 

N/A 

SIl 

Various  users  and/or  customers  are  involved  in  the  requirements  development  and 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

funetionality/feature  identification  process. 

□ 

□ 

□ 

□ 

□ 

□ 

SI2 

Various  user  and/or  customer  concerns  are  specified  and  documented  for  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

project  and  the  product. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

SI3 

Various  user  and/or  customer  profiles  are  identified  and  documented. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

SI4 

Prototypes/user  stories/paper  mock-ups/use  cases  etc.  are  prepared  with  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

involvement  of  users. 

□ 

□ 

□ 

□ 

□ 

□ 

SI5 

Executive/upper  management  is  involved  in  the  decision  making  process  regarding 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

the  project  baselines,  cost  and  schedule  variations  etc. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

SI6 

All  stakeholders  are  identified  and  documented. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

SI7 

There  are  regular  meetings  with  various  stakeholders. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 
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SI8 

There  is  an  information  gathering  activity  to  identify  stakeholders  and 
their  stakes/concerns. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

SI9 

All  stakeholders  show  commitment  to  the  successful  outcome  of  the 
project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

SI10.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  There  is  a  document  guiding  the  management  of  stakeholders. 

□  The  stakeholder  management  plan/document  lists  the  primary  and  secondary  stakeholders. 

□  The  stakeholder  management  plan/document  lists  the  concerns  and  stakes  of  the  primary  and  secondary  stakeholders. 

□  The  stakeholder  management  plan/document  provides  specific  strategies  for  dealing  with  various  stakeholders. 

□  The  users  and/or  customers  participated  in  the  testing  phase  of  the  project. 

□  There  is  a  documented  procedure  for  the  acceptance  of  the  project  deliverables. 

□  None 


*  Respond  to  the  following  questions(SI11-SI12)  only  if  the  project  is  developed  for  the  market  without  a  specific  contract. 


Sill 

The  marketing  department  and  necessary  functional  managers  are  involved  in  the 
decision-making  process  during  development. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

SI12 

The  marketing  department  provides  timely  information  regarding  users  and  other 
competing  products. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

*  Respond  to  the  following  questions  (SI13-SI18)  only  if  the  project  is  developed  under  a  contract  with  a  specific  customer. 


SI13 

There  are  communication  and  coordination  problems  between  project 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

team  members  and  other  stakeholders. 

□ 

□ 

□ 

□ 

□ 

□ 

SI14 

When  there  is  a  change  in  the  baseline,  the  cost,  schedule,  and 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

functionality/features  are  renegotiated  with  the  customer. 

□ 

□ 

□ 

□ 

□ 

□ 

SI15 

Regular  updates  regarding  project  variables  such  as  cost,  schedule  and  progress  on 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

functionality  are  provided  to  the  stakeholders. 

□ 

□ 

□ 

□ 

□ 

□ 

SI16 

When  there  is  an  increase  in  cost  or  delay  in  schedule,  the  news  and  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

consequences  are  shared  with  the  stakeholders  in  a  timely  manner. 

□ 

□ 

□ 

□ 

□ 

□ 

SI17 

Project  milestones  are  considered  reached  when  there  is  consensus  from 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

stakeholders  for  advancing  to  the  next  phase. 

□ 

□ 

□ 

□ 

□ 

□ 

SI18.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

□  Project  team  members  are  allowed  to  have  direct  communication  with  the  customers  and/or  users. 
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□  All  communication  with  the  stakeholders  is  conducted  via  the  project  manager  and/or  management. 

PROJECT  MONITORING  AND  CONTROL  Section  (19  Questions  -  About  4-8  minutes) 

PMC1 .  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

□  There  is  a  documented  project  plan.  □  There  is  no  project  plan. 

PMC2.  Which  of  the  following  data  and/or  metric/s  are  regularly  monitored  and  documented?  (Check  all  that  apply.) 

p  Team/developer  performance 

□  Cost  and  earned  value 

lH  Risk  items  and  their  impacts 

□  Schedule  performance 

□  Number  of  requirements  changes 

l|  Necessary  staff  and  skill  requirements 

□  None 

PMC3.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

□  There  are  specific  project  team  members  assigned  for  controlling  activities  such  as  configuration  management,  requirement  changes  etc. 

□  All  control  activities  are  handled  by  the  project  manager. 

PMC4.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  There  are  project  progress  or  milestone  review  meetings. 

□  Key  project  problems  are  identified  and  being  monitored. 

□  Key  project  problems  and  project  progress  status  is  visible  to  the  stakeholders  including  project  team  members. 

□  None 

PMC5.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  There  is  an  established  requirements  change  and  control  process. 

□  There  is  an  established  risk  management  and  control  process, 
p  There  is  an  established  configuration  management  process. 

□  There  is  an  established  baseline  tracking  and  scope  change  control  process. 

□  There  is  an  established  project  management  data  and  metrics  collection  and  monitoring  process. 

□  None 
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(5) 

(4) 

(3) 

(2) 

(i) 

N/A 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

PMC  6 

The  project  problems  are  generally  proactively  addressed  (before  they  happen). 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

PMC  7 

The  project  problems  are  generally  reaetively  addressed  (when  they  happen). 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

PMC  8 

The  project  resources  are  closely  monitored. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PMC9 

There  is  an  established  project  monitoring  and  control  procedure  with  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

acceptance  of  project  team  members. 

□ 

□ 

□ 

□ 

□ 

□ 

PMC 

There  are  established  methods/criteria  to  determine  deviations  from  the  project 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

10 

plan. 

□ 

□ 

□ 

□ 

□ 

□ 

PMC 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

i  ivi  v 

11 

In  case  of  deviations  from  the  plan,  corrective  action  is  immediately  taken. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PMC 

Project  management  metrics  are  effectively  collected  and  used  in  decision-making. 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

12 

(such  as  planned  versus  actual  cost,  requirements  changes,  schedule  performance 
etc.) 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PMC 

A  project  management  automated  software  tool  is  used  to  manage  project 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

13 

management  data  and  metrics. 

□ 

□ 

□ 

□ 

□ 

□ 

PMC 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

14 

Earned  value  management  is  effectively  used. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PMC 

There  is  communication  between  management  and  project  staff  regarding  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

15 

project  progress  data. 

□ 

□ 

□ 

□ 

□ 

□ 

PMC 

The  commitment  and  concerns  of  various  stakeholders  is  being  monitored  through 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

16 

regular  meetings  and  communication. 

□ 

□ 

□ 

□ 

□ 

□ 

PMC 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

17 

The  subcontractor  performance  is  monitored  regularly. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PMC 

There  are  checklists  for  critical  tasks  such  testing,  version  control,  requirements 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

18 

change  requests  etc. 

□ 

□ 

□ 

□ 

□ 

□ 

PMC 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

19 

Corrective  actions  for  problems  are  timely  and  effective. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 
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PROJECT  PLANNING  AND  ESTIMATION  Section  (35  Questions  -  About  10-18  minutes) 

PPE1.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  There  is  a  formal  documented  project  plan. 

□  There  is  an  informal  project  plan. 

□  There  project  plan  and  schedule  is  made  visual  via  diagrams,  charts  etc. 

□  There  is  no  project  plan. 

PPE2.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

□  The  project  plan  is  developed  as  needed  during  the  project.  □  The  project  plan  is  developed  up  front  before  any  development  effort. 

PPE3.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

Q  The  project  budget,  schedule,  and  staff  requirements  are  strictly  enforced  by  the  executive/upper  management  or  customer. 

□  The  project  budget,  schedule,  and  staff  requirements  are  identified  via  analysis  and  negotiation. 


PPE4.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

□  The  project  plan  is  approved  by  the  stakeholders  such  as  customers,  users,  project  team  members,  executive  management  etc. 

□  There  is  no  approval  process. 

PPE5.  Which  of  the  following/s  is/are  involved  in  the  project  planning?  (Check  all  that  apply.) 

□  Senior/executive/upper  management 

□  Experts  and  consultants 

□  Project  manager  and/or  management  team 

□  Project  team  members 

□  Customer/user/marketing  department 

□  Other  relevant  stakeholders 

□  None 

PPE6.  Which  of  the  following/s  is/are  included  in  the  project  plan?  (Check  all  that  apply.) 

□  Project  scope 

□  Deliverables  or  products  list 

P  Detailed  schedule  and  milestones  /  various  product  version  delivery  dates 

□  Detailed  budget  and  cost  analysis 

□  Staffing/personnel/developer  requirements 

P  Task  responsibility  matrix  or  similar  assignment  matrix 

□  Required  functionality/features  of  the  products  or  deliverables 
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□  Validation  and  verification  plan 

□  Acquisition  plan  /  Subcontracting  planning 

Q  Deployment  or  Installation  plan/  Marketing  plan 

□  Quality  requirements  /  Quality  assurance  plan 

□  Risk  management  planning 
[H  Project  glossary 

□  Project  communications  planning 

□  Project  organization  charts 

Q  Staff  responsibilities  and  responsibility  definitions 

□  Necessary  facility,  equipment,  and  component  requirements 

□  None 

PPE7.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  There  is  a  statement  of  work  (or  a  similar  document)  stating  what  needs  to  be  accomplished/done. 

□  There  is  a  work  breakdown  structure  or  a  feature/functionality  list  (or  a  similar  document)  that  details  the  project  tasks/activities. 

□  The  tasks  and  activities  are  identified  as  the  project  progresses. 

□  None 

PPE8.  What  kinds  of  effort,  schedule  or  cost  estimation  techniques  are  used?  (Check  all  that  apply.) 

P  Experiences  of  project  manager/management  team 

□  Inputs  from  project  team  members 

□  Expert  or  consultant  judgment 

□  Analogy  to  similar  projects 
P  Historical  data 

□  Automated  cost  estimation  tools 

□  None 

PPE9.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  No  estimation  is  needed. 

□  Only  one  type  of  estimation  technique  is  used. 

□  Two  or  more  estimation  techniques  are  used. 

□  Estimates  from  various  techniques  are  compared  and  analyzed  for  discrepancies. 

□  None 

PPE10.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  Lines  of  code  (LOC)  are  used  in  estimation. 

□  Function  points  are  used  in  estimation. 
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p  Number  of  functionality/features  are  used  in  estimation, 
p  Number  of  modules  and  deliverables  are  used  in  estimation. 

□  Other  advanced  metrics  used  in  estimation. 

□  None 


(5) 

(4) 

(3) 

(2) 

(i) 

N/A 

PPE 

11 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

The  project  schedule  is  feasible. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PPE 

12 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

The  funding  for  the  project  is  adequate. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PPE 

13 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

The  project  is  adequately  staffed. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PPF 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

r  iej 

14 

Extra  funding  for  unprecedented  issues  is  set  aside. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PPE 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

15 

Slack  or  buffer  time  exists  in  the  schedule  for  unprecedented  or  extra  activities. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PPE 

Alternative  staff  to  accomplish  critical  tasks/aetivities  are  considered  and 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

16 

incorporated  in  the  project  plan. 

□ 

□ 

□ 

□ 

□ 

□ 

PPE 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

17 

All  relevant  stakeholders  are  identified  before  planning  activities. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PPE 

A  certain  level  of  requirements  analysis  is  conducted  before  planning  and 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

18 

estimation. 

□ 

□ 

□ 

□ 

□ 

□ 

PPF 

All  external  dependencies  are  identified  and  incorporated  to  the  planning.  (Such  as 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

rrt 

19 

acquisition  of  various  products  and  services  from  outside  vendors,  required 
permissions  from  various  authorities,  etc.) 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PPF 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

rrt 

20 

The  project  plan  is  updated  throughout  the  project  development. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PPE 

The  project  plan  is  visible/available  to  project  team  members  and  other 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

21 

relevant  stakeholders. 

□ 

□ 

□ 

□ 

□ 

□ 

PPE 

Various  automated  project  management  tools  are  used  in  planning  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

22 

project. 

□ 

□ 

□ 

□ 

□ 

□ 
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PPE 

The  project  team  members  are  consulted  in  planning  and  estimation 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

23 

efforts. 

□ 

□ 

□ 

□ 

□ 

□ 

PPE 

The  managers  at  various  levels  have  project  planning  and  estimation 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

24 

training. 

□ 

□ 

□ 

□ 

□ 

□ 

PPE 

Each  task/activities/work  packages  are  assigned  to  specific  project  team 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

25 

member  or  members. 

□ 

□ 

□ 

□ 

□ 

□ 

PPE 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

1  1  L 

26 

Critical  activities  are  identified  and/or  critical  path  analysis  is  conducted. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PPE 

Various  standards,  guidelines  or  checklists  are  used  in  planning  and 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

27 

estimation. 

□ 

□ 

□ 

□ 

□ 

□ 

PPE 

Formal  analysis  is  conducted  for  cost,  schedule  and  effort  estimation  such 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

28 

as  PERT,  CPM  etc. 

□ 

□ 

□ 

□ 

□ 

□ 

PPE 

Factors  such  as  staff  turnover  or  loss  of  key  personnel  are  considered 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

29 

during  planning. 

□ 

□ 

□ 

□ 

□ 

□ 

PPE 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

1  1  L 

30 

Realistic  estimates  guide  the  project  planning. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PPE 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

1  1  L 

31 

Testing  is  carefully  incorporated  to  project  plan. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PPE 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

1  1  L 

32 

Effort  estimations  are  provided  by  those  performing  the  tasks. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PPE 

Project  risks  are  carefully  analyzed  and  contingencies  are  included  in  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

33 

planning. 

□ 

□ 

□ 

□ 

□ 

□ 

PPE 

A  suitable  project  development  approach  and  process  is  identified  with 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

34 

rationale  in  the  project  plan. 

□ 

□ 

□ 

□ 

□ 

□ 

PPE 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

1  1  L 

35 

All  necessary  skills  and  expertise  needed  in  the  project  are  identified. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 
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SCOPE  MANAGEMENT  Section  (16  Questions  -  About  3-8  minutes) 

SMI.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

□  Project  scope  never  changed.  □  Project  scope  frequently  changed.  □  Project  scope  somewhat  changed. 

SM2.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

Q  Project  scope  is  ambiguous  at  first  and  it  becomes  clear  during  the  project. 

□  Project  scope  is  ambiguous  at  first  and  stays  ambiguous  due  to  various  reasons. 

□  Project  scope  is  defined  and  clear  at  the  beginning  of  the  project  and  it  stays  clear. 

□  Project  scope  is  defined  and  clear  at  the  beginning  of  the  project  and  it  become  ambiguous  due  to  various  reasons. 

SM3.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

□  There  is  a  project  scope  document  and  it  stayed  the  same  from  the  project  start. 

□  There  is  a  project  scope  document  and  it  is  updated  when  it  is  necessary. 

□  There  isn’t  a  project  scope  document. 

SM4.  What  is  the  effect  of  project  scope  changes  on  the  project  schedule?  (Check  only  one.) 

□  None  □  On  time  without  scope  change/s  □  On  time  with  scope  change/s  Q  Late  without  scope  change/s 

□  Late  with  scope  change/s 

SM5.  What  is  the  effect  of  project  scope  changes  on  the  project  budget?  (Check  only  one.) 

□  None 

□  Within  budget  without  scope  change/s 

□  Within  budget  with  scope  change/s 

□  Cost  overrun  without  scope  change/s 

□  Cost  overrun  with  scope  change/s 

SM6.  What  is  the  effect  of  project  scope  changes  on  the  functionality  of  the  deliverables?  (Check  only  one.) 

□  None 

□  Full  functionality  without  scope  change/s 

□  Full  functionality  with  scope  change/s 

□  Less  than  planned  functionality  without  scope  change/s 

□  Less  than  planned  functionality  with  scope  change/s 

SM7.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  Project  scope  changes  are  handled  only  by  the  management. 

□  Project  scope  changes  have  to  follow  a  formal  defined  process. 
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P  Project  scope  changes  follow  a  decision-making  process  that  includes  management,  stakeholders,  and  team  members. 

□  Project  scope  changes  handled  informally  by  the  management. 

SM8.  Which  of  the  following  statement/s  is/are  included  in  the  project  scope  document,  if  there  is  one.  (Check  all  that  apply.) 

□  The  problem  statement 

□  The  work  to  be  done  or  work  breakdown  structure 

□  The  constraints 

□  The  resources 

□  Preliminary  or  detailed  schedule  and  cost  analysis 

□  The  project  deliverables 

□  Clear  definition  of  performance  to  meet  contractual  and  legal  obligations 

□  Glossary 

□  Not  Available 

SM9.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  The  project  scope  is  defined  after  stakeholders  are  identified. 

□  There  is  at  least  one  project  scope  identification/definition  meeting  at  the  beginning  of  the  project. 

□  There  is  a  project  scope  change  board. 

SM10.  Who  are  included  while  defining  and  updating  the  project  scope?  (Check  all  that  apply.) 

□  Project  management  team 

□  Project  manager 

□  All  stakeholders 

□  Some  stakeholders 

□  Project  team  members 

□  Subcontractor  representatives  if  there  is  subcontracting 

□  None 


133 


(5) 

(4) 

(3) 

(2) 

(i) 

N/A 

SMll 

Before  defining  the  project  scope,  there  is  a  rigorous  information  gathering 
activity  about  the  problem  that  is  to  be  solved,  the  resources,  the 
constraints,  the  deliverables  etc. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

SM12 

Project  scope  is  not  clearly  defined  due  to  various  reasons. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

SM13 

The  project  has  a  documented  project  scope  definition  and  a  formal  scope 
change  process. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

SMI  4 

Project  scope  is  always  visible  and  clear  to  stakeholders,  project  team  members, 
and  management. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

SM15 

Project  scope  changes  have  to  go  through  an  extensive  decision-making  process. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

SM16 

The  project  scope  document  is  reviewed  and  approved  by  all  stakeholders. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RISK  CONTROL  Section  (17  Questions  -  About  3-8  minutes) 

RC1.  What  is  the  overall  risk  level  of  the  project?  (Check  only  one.) 

□  High  □  Medium  □  Low  □  None 

RC2.  What  is  the  effect  of  risks  on  the  project  budget?  (Check  only  one.) 

□  High  cost  overrun  □  Medium  cost  overrun^  Low  cost  overrun  □  None 

RC3.  What  is  the  effect  of  risks  on  the  project  schedule?  (Check  only  one.) 

□  The  project  delivery  is  on  time.  □  The  project  delivery  is  slightly  late.  □  The  project  delivery  is  significantly  late. 

RC4.  What  is  the  effect  of  risks  on  the  project  functionality?  (Check  only  one.) 

□  High  □  Medium  □  Low  □  None 

RC5.  What  is  the  level  of  funding  and  resources  set  aside  for  risk  management?  (Check  only  one.) 

□  More  than  enough  □Enough  □  Hardly  enough  □  No  funding  and  resources 

RC6.  Check  the  statement/s  that  applies  to  the  project.  (Check  only  one.) 

□  Adequate  slack  time  is  planned  in  the  schedule  for  consequences  due  to  risks. 
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P  There  is  not  any  slack  time  planned  for  consequences  due  to  risks. 

□  Not  enough  slack  time  is  planned  in  the  schedule  for  consequences  due  to  risks. 

RC7.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

□  Risks  are  handled  when  they  occur.  □  Risks  are  addressed  before  they  occur.  □  Both 

RC8.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  Informal  project  risk  management  procedures  are  in  place. 

□  Project  risk  management  is  based  on  formal  procedures. 

□  There  is  not  any  project  risk  management  and  planning. 

RC9.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  Risks  are  generally  avoided.  (Risk  Avoidance) 

□  Risks  are  transferred  to  third  parties  for  example  contracting  risky  development  items  to  consultants  or  experts.  (Risk  Transfer) 

□  Risks  are  managed  as  they  occur. 

□  Risk  mitigation  (actions  reducing  the  severity/impact  of  a  risk)  is  the  most  used  option  in  risk  management  of  the  project.  (Risk  Mitigation) 

□  None 

RC10.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

□  Experts  are  consulted  in  the  risk  management  of  the  project. 

□  Project  management  handles  all  the  risks. 

□  Project  team  members  and  stakeholders  are  involved  in  the  risk  management. 


(5) 

(4) 

(3) 

(2) 

(i) 

N/A 

RCll 

For  each  identified  risk  item,  there  is  an  information  gathering  activity. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RC12 

Contingencies  and  alternative  solutions  are  planned  for  the  critical  tasks 
and  portions  of  the  development  exposed  to  high  risks. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RC13 

Top  risk  items  list  is  closely  monitored  and  periodically  updated. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RC14 

Risk  monitoring  is  an  important  activity  in  the  project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RC15 

Risk  avoidance  is  primary  method  of  risk  control  activities. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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RC16 

There  are  regular  project  risk  monitoring  meetings  or  project  risk  monitoring  is 
handled  through  project  status  meetings  etc. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RC17 

There  is  a  risk  management  plan  and  course  of  action  for  each  high-risk  items. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

STAFFING/HIRING  Section  (29  Questions  -  About  7-13  minutes) 

51.  Which  of  the  followings  are  clearly  identified,  documented  and  communicated?  (Check  all  that  apply.) 

□  Project  Roles  □  Project  Positions  □  Necessary  Qualifications  for  the  project  □  None 

52.  Which  of  the  documents  or  similar  documents  exist  for  the  project?  (Check  all  that  apply.) 

□  Project  staffing  management  plan 

Q  Project  responsibility/accountability/interfaces/assignment  matrix 
O  Project  work  breakdown  structure 

□  None 

53.  What  is  the  experienced-to-inexperienced  project  team  member  ratio?  (experienced:  inexperienced)  (Check  only  one.) 

□  Smaller  than  1:2  □  1:2  □  1:1  Q2:1  □  Greater  than  2:1 

54.  Which  of  the  followings  for  team  members  are  clearly  identified,  documented  and  communicated?  (Check  all  that  apply.) 

□  Responsibility  □  Job  Interfaces  □  Reporting  Structure  □  None 


(5) 

(4) 

(3) 

(2) 

(i) 

N/A 

S5 

The  work  breakdown  structure  (WBS)  or  similar  document  is  completed 
before  hiring/staffing. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S6 

The  analysis  of  the  required  work  and  resources  is  conducted  rigorously. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S7 

Significant  project  risks  are  identified  before  the  hiring/staffing  the  team. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S8 

There  is  adequate  funding  and  resources  for  hiring/staffing. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S9 

There  are  adequate  work  force  and  experts  with  the  necessary  skills  and  expertise 
available  for  hiring  and/or  staffing  on  this  project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

S10 

Expertise  on  human  resources  is  acquired  for  staffing  and  hiring  activities. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

S11 

Project  open  positions  are  made  attractive  to  qualified  candidates  through 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

incentives  etc.  The  position  is  made  desirable. 

□ 

□ 

□ 

□ 

□ 

□ 

S12 

The  skills  and  expertise  needed  for  the  project  success  are  acquired  with  the  timely 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

recruitment  of  team  members. 

□ 

□ 

□ 

□ 

□ 

□ 

S13 

The  necessary  interpersonal  skills  for  the  roles  are  identified  and  the  project  team 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

members  are  recruited  also  based  on  their  interpersonal  skills. 

□ 

□ 

□ 

□ 

□ 

□ 

S14 

The  ambitions  and  goals  of  the  project  team  members  are  aligned  with  the  project 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

mission  and  goals. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

S15 

The  project  team  members  have  the  necessary  educational  background. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

S16 

The  project  team  members  have  similar  project  work  experience. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

S17 

The  productivity  of  the  project  team  members  are  within  the  expectations. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

S18 

Project  team  members  are  familiar  and  comfortable  with  the  organizational 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

culture. 

□ 

□ 

□ 

□ 

□ 

□ 

S19 

Project  team  members  have  difficulties  with  the  organizational 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

procedures. 

□ 

□ 

□ 

□ 

□ 

□ 

S20 

Project  team  members  are  happy  with  their  roles,  positions  and  career 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

advancement  opportunities  in  the  project  organization. 

□ 

□ 

□ 

□ 

□ 

□ 

S21 

Project  team  members  stay  with  the  project  according  to  the  project  staffing 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

management  plan.  Turn-over  rate  is  at  minimum. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

S22 

Resignations  are  at  minimum. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

S23 

Project  team  members  acquire  the  necessary  skills  and  expertise  needed  for  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

project  through  training  and  coaching. 

□ 

□ 

□ 

□ 

□ 

□ 
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S24 

There  are  alternative  team  members  with  the  necessary  skills  and  knowledge  to 
take  over  some  other  team  member’s  work  for  critical  tasks  in  case  of  team 
member  loss. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S25 

Project  positions  are  filled  with  qualified  individuals. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S26 

Work  and  task  assignments  are  fair  and  based  on  qualifications  of  the  project  team 
members. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S27 

Removing  of  project  team  members  for  unsatisfactory  work  performance  and/or 
other  reasons  are  conducted  fairly  and  according  to  the  organizational  procedures. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S28 

Orientation  or  transition  activities  for  the  new  team  members  are  conducted 
properly. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S29 

When  necessary,  consultants  and  contractors  are  used  effectively. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

CONFIGURATION  MANAGEMENT  Section  (13  Questions  -  About  3-7  minutes) 

*  In  some  organizations,  configuration  management  is  referred  to  as  version  control. 

CM1 .  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

□  Configuration  management  is  conducted  informally. 

□  Configuration  management  is  a  formal  and  documented  activity  and  it  has  well-defined  procedures. 

CM2.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  There  is  a  configuration  management  document. 

□  There  is  a  configuration  or  change  control  board,  committee  or  team. 

□  There  is  a  configuration  items  list. 

□  None 

CM3.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  Baselines  and  configuration  items  are  identified  at  the  beginning  of  the  project  and  updated  as  necessary. 

□  The  owner  or  responsible  staff  is  identified  for  each  configuration  item. 

P  Every  configuration  item  has  a  unique  identifier. 

□  Important  characteristics  for  each  configuration  item  are  identified  such  as  author,  type,  date,  version  number  etc. 
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□  None 


CM4.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  The  configuration  management  procedures  includes  a  detailed  change  and  change  request  protocols. 

Q  The  configuration  management  system  has  various  levels  of  control  (such  as  only  author  may  release  the  item,  restricted  write  access  etc). 

□  There  is  not  a  configuration  management  system  and  configuration  management  is  only  the  responsibility  of  project  team  members  or 
developers. 

□  None 

CM5.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

□  The  change  requests  have  to  go  through  the  change  control  board  or  responsible  staff. 

□  The  change  requests  are  only  handled  by  the  developer  or  the  owner  of  the  configuration  item. 


(5) 

(4) 

(3) 

(2) 

(i) 

N/A 

CM6 

The  project  suffers  from  configuration/version  management  problems. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

CM7 

An  automated  configuration  management  system  is  used  and  adequate  for 
the  project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

CM8 

The  configuration  management  procedures  are  strictly  followed.  Project  team 
members  do  not  try  to  bypass  them. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

CM9 

The  integrity,  security  and  privacy  of  configuration  items  are  satisfactory. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

CM10 

The  changes  and  change  requests  are  controlled,  and  documented  in  such  a  way 
that  it  enables  audit. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

CM11 

Every  change  request  is  controlled  and  extensively  reviewed. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

CM12 

Records  of  configuration  management  activities,  changes  to  baselines,  work 
products,  and  change  requests  are  well-maintained. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

CM13 

There  is  an  established  and  reliable  configuration  management  system  including 
automated  tools,  databases,  protocols  etc. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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RISK  ASSESSMENT  Section  (20  Questions  -  About  5-10  minutes) 

RA1.  Which  of  the  following  does  best  characterize  the  risk  assessment  activities  in  the  project?  (Check  only  one.) 

□  Formal  □  Informal  □  Semiformal  □  Not  available 

RA2.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  Risks  are  assessed  as  they  are  identified  during  the  project. 

□  Risks  are  assessed  early  and  incorporated  into  a  risk  management  document. 

□  The  risk  management  document  is  periodically  updated. 

□  There  is  staff  specifically  assigned  to  risk  assessment  activities. 

□  Lessons  learned  are  visited  prior  to  risk  assessment  activities. 

□  None 

RA3.  In  which  of  the  following  categories  the  risks  are  assessed  and  documented?  (Check  all  that  apply.)  _ 

□  People  □  Schedule  □  Budget  and  Funding  □  Technology  □Requirements  □Subcontractor  LJ 

None 

RA4.  There  are  common  objective  criteria  to  assess  risks.  (Check  only  one.) 

□  Yes  □  No  □  Partially  □  Not  Available 

RA5.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  There  is  a  project  risk  management  plan. 

□  The  project  risk  management  plan  includes  objective  criteria  for  risk  identification,  analysis  and  prioritization. 

□  Project  risk  document  is  updated  frequently  along  the  project. 

□  None 

RA6.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  Experts  or  consultants  are  used  for  risk  assessment. 

□  Experienced  project  staff  is  used  for  risk  assessment. 

□  Project  manager  conducted  the  risk  assessment. 

□  There  is  not  any  risk  assessment  activity. 

RA7.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  Risks  are  identified.  □  Risks  are  analyzed.  □  Risks  are  categorized.  □  Risks  are  prioritized.  □  None 

RA8.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

□  Risk  assessment  is  based  on  qualitative  methods. 
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□  Risk  assessment  is  based  on  quantitative  methods. 

□  Risk  assessment  is  based  on  the  judgment  of  the  management. 

Q  Risk  assessment  is  based  on  both  qualitative  and  quantitative  methods. 

□  There  is  no  need  for  any  risk  assessment  activity. 


(5) 

(4) 

(3) 

(2) 

(i) 

N/A 

RA9 

The  projects  risks  are  documented  early  with  details  related  to  their  impact 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

on  the  project. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RAID 

Risk  assessment  has  a  clear  impact  on  project  planning  and  decisions. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

RA11 

Sufficient  reserve  resources  and  funding  are  planned  and  set  aside  for  risk 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

assessment  activities. 

□ 

□ 

□ 

□ 

□ 

□ 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RA12 

Top  risk  items  list  or  a  similar  list  is  maintained. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

RA13 

Risks  are  assessed  with  the  broad  inclusion  of  stakeholders  and  project  team 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

members. 

□ 

□ 

□ 

□ 

□ 

□ 

RA14 

Project  environment  facilitates  and  encourages  open  and  free  discussions  on  project 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

risks. 

□ 

□ 

□ 

□ 

□ 

□ 

RA15 

Risks  are  identified  using  risk  identification  tools  such  as  checklists,  databases,  risk 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

taxonomy,  decision-driver  analysis,  etc. 

□ 

□ 

□ 

□ 

□ 

□ 

RA16 

Risks  are  analyzed  based  on  their  probability  of  occurrence  and  impact  on  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

project. 

□ 

□ 

□ 

□ 

□ 

□ 

RA17 

Risks  are  prioritized  based  on  their  probability  of  occurrence  and  impact  on  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

project. 

□ 

□ 

□ 

□ 

□ 

□ 

RA18 

Risk  assessment  information  is  always  visible  and  they  are  shared  with  stakeholders 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

and  project  team  members. 

□ 

□ 

□ 

□ 

□ 

□ 

RA19 

Any  stakeholder  or  project  team  member  may  report  a  risk  at  any  time  and  there  is 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

a  mechanism  allowing  such  reports. 

□ 

□ 

□ 

□ 

□ 

□ 
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RA20.  (Answer  only  if  a  portion  of  the  system  is  subcontracted.)  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  Subcontractor/s  is/are  free  in  their  risk  management  decision  and  activities. 

□  Subcontractor/s  is/are  contractually  responsible  to  have  formal  risk  assessment  procedures. 

□  Subcontractor/s  is/are  contractually  responsible  to  deliver  risk  assessment  reports. 

□  Subcontractor/s  has/have  a  representative  for  project  risk  management  meetings. 

QUALITY  ENGINEERING  Section  (20  Questions  -  About  4-10  minutes) 

QE1.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  There  is  a  quality  policy. 

□  Quality  is  not  a  high  priority  in  this  project  due  to  various  reasons. 

□  There  is  a  quality  planning  activity. 

QE2.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

P  Quality  expectations  of  various  stakeholders  are  identified  and  documented. 

□  The  quality  standards  and  guidelines  related  to  the  project  are  identified.  (Such  as  aviation  standards  etc.) 
p  Objective  quality  criteria  for  the  project  and  its  deliverables  are  identified. 

□  None 

QE3.  Which  of  the  following  quality  attribute/s  are  considered  achieved  in  the  project?  (Check  all  that  apply.) 

□  Maintainability  □  Safety  □  Security  □  Reliability  □  Usability  □  Other  □  None 

QE4.  What  is  the  amount  of  testing  conducted  during  the  project  development?  (Check  only  one.) 

□  Extensive  □  Fair  □  Some  □  None 


(5) 

(4) 

(3) 

(2) 

(i) 

N/A 

QE5 

Quality  is  considered  a  high  priority  in  this  project. 

Completely 

Agree 

n 

Agree 

n 

Neutral 

n 

Disagree 

n 

Completely 

Disagree 

n 

Not 

Applicable 

n 

QE6 

There  is  support  for  and  commitment  to  quality  from  executive  management. 

Completely 

Agree 

n 

Agree 

n 

Neutral 

n 

Disagree 

n 

Completely 

Disagree 

n 

Not 

Applicable 

n 

QE7 

High  quality  is  planned  from  the  start  in  this  project. 

Completely 

Agree 

n 

Agree 

n 

Neutral 

n 

Disagree 

n 

Completely 

Disagree 

n 

Not 

Applicable 

n 

QE8 

Various  quality  metrics  are  identified. 

Completely 

Agree 

_n _ 

Agree 

_n _ 

Neutral 

_ 

Disagree 

_n _ 

Completely 

Disagree 

_n _ 

Not 

Applicable 

□ 
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Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE9 

Quality  assurance  procedures  are  adequate. 

Agree 

n 

n 

n 

n 

Disagree 

n 

Applicable 

n 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE10 

Quality  assurance  procedures  are  documented. 

Agree 

n 

n 

n 

n 

Disagree 

n 

Applicable 

n 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE11 

Adequate  amount  of  resources  are  set  aside  for  quality  engineering  activities. 

Agree 

n 

n 

n 

n 

Disagree 

n 

Applicable 

n 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE12 

The  requirements  are  defined  with  the  guidance  of  quality  expectations. 

Agree 

n 

n 

n 

n 

Disagree 

n 

Applicable 

n 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE13 

The  project  team  culture  encourages  commitment  to  high  quality. 

Agree 

n 

n 

n 

n 

Disagree 

n 

Applicable 

n 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE14 

Project  team  members  are  trained  in  quality  assurance. 

Agree 

n 

n 

n 

n 

Disagree 

n 

Applicable 

n 

QE15 

There  are  quality  thresholds  and  expectations  for  various  work  products  such  as 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

system  architecture,  requirements  definitions,  designs,  testing  etc. 

n 

n 

n 

n 

n 

n 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE16 

Quality  considerations  are  limited  to  testing. 

Agree 

n 

n 

n 

n 

Disagree 

n 

Applicable 

n 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE17 

High  testing  coverage  for  the  product  is  achieved. 

Agree 

n 

n 

n 

n 

Disagree 

n 

Applicable 

n 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE18 

There  are  adequate  tools,  equipment,  and  resources  fortesting. 

Agree 

n 

n 

n 

n 

Disagree 

n 

Applicable 

n 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE19 

There  are  specifically  assigned  team  members  for  quality  related  issues. 

Agree 

_n _ 

_n _ 

_ 

_n _ 

Disagree 

_n _ 

Applicable 

□ 

QE20.  Which  of  the  following  activity  or  activities  are  conducted  during  the  project  development?  (Check  all  that  apply.) 

□  Design  reviews 

□  Code  reviews/inspections 

□  Performance  testing 

□  Independent  verification  and  validation 

□  Quality  assurance  activities 

□  Requirements  tracing 

□  Various  types  of  testing 

□  Defect  identification  and  prevention 

□  Simulations  and/or  prototyping 

□  None 
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Thank  you  very  much  for  your  time  and  participation.  After  completing  the  questionnaire,  please  respond  the 
following  question  again.  It  does  not  have  to  be  the  same  as  your  initial  assessment. 


After  you  completed  the  questionnaire,  how  would  you  rate  the  overall  success  of  the  project? 
(0  being  a  complete  failure  and  10  being  a  complete  success.) 


PR18. 


0123456789  10 

□  □□  □□  □□  □□□□ 
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APPENDIX  C.  SOFTWARE  PROJECT  MANAGEMENT 
EVALUATION  INSTRUMENT  SCORES 


Question  Number 

A 

B 

C 

D 

E 

G 

Cl 

2 

2 

2 

2 

2 

0 

C2 

1 

1 

1 

1 

1 

1 

C3 

1 

1 

1 

1 

1 

1  0 

C4 

1 

1 

1 

1 

1 

1  0 

Question  Number 

SA 

A 

N 

D 

SD 

NA 

C5 

2 

1 

0 

-1 

-2 

0 

C6 

2 

1 

0 

-1 

-2 

0 

C7 

-2 

-1 

0 

1 

2 

0 

C8 

2 

1 

0 

-1 

-2 

0 

C9 

2 

1 

0 

-1 

-2 

0 

CIO 

2 

1 

0 

-1 

-2 

0 

Cll 

2 

1 

0 

-1 

-2 

0 

C12 

2 

1 

0 

-1 

-2 

0 

C13 

2 

1 

0 

-1 

-2 

0 

C14 

2 

1 

0 

-1 

-2 

0 

C15 

2 

1 

0 

-1 

-2 

0 

C16 

2 

1 

0 

-1 

-2 

0 

C17 

-2 

-1 

0 

1 

2 

0 

C18 

2 

1 

0 

-1 

-2 

0 

C19 

2 

1 

0 

-1 

-2 

0 

C20 

2 

1 

0 

-1 

-2 

0 

C21 

2 

1 

0 

-1 

-2 

0 

C22 

2 

1 

0 

-1 

-2 

0 

C23 

2 

1 

0 

-1 

-2 

0 
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Question  Number 

A 

B 

C 

D 

E 

G 

T1 

1 

1 

1 

1 

1 

0 

T2 

2 

0 

-1 

-2 

T3 

2 

2 

2 

2 

0 

T4 

1 

1 

1 

1 

1 

1  0 

Question  Number 

SA 

A 

N 

D 

SD 

NA 

T5 

2 

1 

0 

-1 

-2 

0 

T6 

2 

1 

0 

-1 

-2 

0 

T7 

2 

1 

0 

-1 

-2 

0 

T8 

2 

1 

0 

-1 

-2 

0 

T9 

2 

1 

0 

-1 

-2 

0 

T10 

2 

1 

0 

-1 

-2 

0 

Til 

2 

1 

0 

-1 

-2 

0 

T12 

2 

1 

0 

-1 

-2 

0 

T13 

2 

1 

0 

-1 

-2 

0 

T14 

2 

1 

0 

-1 

-2 

0 

T15 

2 

1 

0 

-1 

-2 

0 

T16 

2 

1 

0 

-1 

-2 

0 

T17 

2 

1 

0 

-1 

-2 

0 

T18 

2 

1 

0 

-1 

-2 

0 

T19 

2 

1 

0 

-1 

-2 

0 

T20 

2 

1 

0 

-1 

-2 

0 

T21 

2 

1 

0 

-1 

-2 

0 

T22 

-2 

-1 

0 

1 

2 

0 

T23 

2 

1 

0 

-1 

-2 

0 

T24 

2 

1 

0 

-1 

-2 

0 

T25 

2 

1 

0 

-1 

-2 

0 

T26 

2 

1 

0 

-1 

-2 

0 

T27 

2 

1 

0 

-1 

-2 

0 

T28 

-2 

-1 

0 

1 

2 

0 

T29 

-2 

-1 

0 

1 

2 

0 

T30 

2 

1 

0 

-1 

-2 

0 

146 


Question  Number  SA 

A 

N 

D 

SD 

NA 

LI 

-2 

-1 

0 

1 

2 

0 

L2 

2 

1 

0 

-1 

-2 

0 

L3 

2 

1 

0 

-1 

-2 

0 

L4 

2 

1 

0 

-1 

-2 

0 

L5 

2 

1 

0 

-1 

-2 

0 

L6 

2 

1 

0 

-1 

-2 

0 

L7 

2 

1 

0 

-1 

-2 

0 

L8 

2 

1 

0 

-1 

-2 

0 

L9 

2 

1 

0 

-1 

-2 

0 

L10 

2 

1 

0 

-1 

-2 

0 

Lll 

-2 

-1 

0 

1 

2 

0 

L12 

-2 

-1 

0 

1 

2 

0 

L13 

2 

1 

0 

-1 

-2 

0 

L14 

2 

1 

0 

-1 

-2 

0 

L15 

2 

1 

0 

-1 

-2 

0 

L16 

2 

1 

0 

-1 

-2 

0 

Question  Number  A 

B 

C 

D 

LI  7* 

2 

2 

1 

-2 

L18* 

-2 

1 

2 

2 

147 


Question  Number  SA 

A 

N 

D  SD 

NA 

OC1 

2 

1 

0 

-1 

-2 

0 

OC2 

2 

1 

0 

-1 

-2 

0 

OC3 

2 

1 

0 

-1 

-2 

0 

OC4 

2 

1 

0 

-1 

-2 

0 

OC5 

2 

1 

0 

-1 

-2 

0 

OC6 

2 

1 

0 

-1 

-2 

0 

OC7 

2 

1 

0 

-1 

-2 

0 

OC8 

2 

1 

0 

-1 

-2 

0 

OC9 

2 

1 

0 

-1 

-2 

0 

OCIO 

2 

1 

0 

-1 

-2 

0 

OC11 

2 

1 

0 

-1 

-2 

0 

OC12 

2 

1 

0 

-1 

-2 

0 

OC13 

2 

1 

0 

-1 

-2 

0 

OC14 

2 

1 

0 

-1 

-2 

0 

OC15 

2 

1 

0 

-1 

-2 

0 

OC16 

2 

1 

0 

-1 

-2 

0 

OC17 

2 

1 

0 

-1 

-2 

0 

OC18 

-2 

1 

0 

-1 

-2 

0 

OC19 

-2 

1 

0 

-1 

-2 

0 

OC20 

2 

1 

0 

-1 

-2 

0 

OC21 

2 

1 

0 

-1 

-2 

0 

OC22 

2 

1 

0 

-1 

-2 

0 

OC23 

2 

1 

0 

-1 

-2 

0 

OC24 

2 

1 

0 

-1 

-2 

0 

Question  Number  A 

B 

|C 

D  E 

F 

OC25 

2 

2 

2 

0 

OC26 

2 

2 

2 

-2 

2 

0 

148 


Question  Number 


C 


E 


I 


PM1 

0 

-2 

-4 

-6 

PM2 

0 

1 

2 

3 

4 

PM3 

1 

1 

1 

1 

1 

0 

PM4 

1 

1 

1 

1 

1 

0 

Question  Number  SA 

A 

N 

D 

SD 

NA 

PM5 

2 

1 

0 

-1 

-2 

0 

PM6 

2 

1 

0 

-1 

-2 

0 

PM7 

2 

1 

0 

-1 

-2 

0 

PM8 

2 

1 

0 

-1 

-2 

0 

PM9 

2 

1 

0 

-1 

-2 

0 

PM10 

2 

1 

0 

-1 

-2 

0 

PM11 

2 

1 

0 

-1 

-2 

0 

PM12 

2 

1 

0 

-1 

-2 

0 

PM13 

2 

1 

0 

-1 

-2 

0 

PM14 

2 

1 

0 

-1 

-2 

0 

PM15 

2 

1 

0 

-1 

-2 

0 

PM16 

2 

1 

0 

-1 

-2 

0 

PM17 

2 

1 

0 

-1 

-2 

0 

PM18 

2 

1 

0 

-1 

-2 

0 

PM19 

2 

1 

0 

-1 

-2 

0 

PM20 

2 

1 

0 

-1 

-2 

0 

PM21 

2 

1 

0 

-1 

-2 

0 

PM22 

2 

1 

0 

-1 

-2 

0 

PM23 

2 

1 

0 

-1 

-2 

0 

PM24 

2 

1 

0 

-1 

-2 

0 

PM25 

2 

1 

0 

-1 

-2 

0 

PM26 

2 

1 

0 

-1 

-2 

0 

PM27 

2 

1 

0 

-1 

-2 

0 

149 


Question  Number  A 


C 


E 


i 


RM1 

2 

2 

2 

2 

0 

RM2 

-2 

2 

2 

-2 

0 

RM3 

1 

1 

1 

1 

1 

0 

RM4 

2 

2 

2 

2 

2 

0 

Question  Number  SA 

A 

N 

D 

SD 

NA 

RM5 

2 

1 

0 

-1 

-2 

0 

RM6 

2 

1 

0 

-1 

-2 

0 

RM7 

2 

1 

0 

-1 

-2 

0 

RM8 

2 

1 

0 

-1 

-2 

0 

RM9 

2 

1 

0 

-1 

-2 

0 

RM10 

2 

1 

0 

-1 

-2 

0 

RM11 

2 

1 

0 

-1 

-2 

0 

RM12 

2 

1 

0 

-1 

-2 

0 

RM13 

2 

1 

0 

-1 

-2 

0 

RM14 

2 

1 

0 

-1 

-2 

0 

RM15 

2 

1 

0 

-1 

-2 

0 

RM16 

2 

1 

0 

-1 

-2 

0 

RM17 

2 

1 

0 

-1 

-2 

0 

RM18 

2 

1 

0 

-1 

-2 

0 

RM19 

2 

1 

0 

-1 

-2 

0 

RM20 

2 

1 

0 

-1 

-2 

0 

RM21 

2 

1 

0 

-1 

-2 

0 

RM22 

2 

1 

0 

-1 

-2 

0 

RM23 

2 

1 

0 

-1 

-2 

0 

RM24 

2 

1 

0 

-1 

-2 

0 

RM25 

2 

1 

0 

-1 

-2 

0 

RM26 

-2 

1 

0 

-1 

-2 

0 

RM27 

2 

1 

0 

-1 

-2 

0 

150 


Question  Number 

SA 

A 

N 

D 

SD 

NA 

Sll 

2 

1 

0 

-1 

-2 

0 

SI  2 

2 

1 

0 

-1 

-2 

0 

SI  3 

2 

1 

0 

-1 

-2 

0 

SI  4 

2 

1 

0 

-1 

-2 

0 

SI  5 

2 

1 

0 

-1 

-2 

0 

SI  6 

2 

1 

0 

-1 

-2 

0 

SI  7 

2 

1 

0 

-1 

-2 

0 

SI  8 

2 

1 

0 

-1 

-2 

0 

SI  9 

2 

1 

0 

-1 

-2 

0 

Question  Number 

A 

B 

C 

D 

E 

G 

SI  10 

2 

2 

2 

2 

2 

2  0 

Sill 

2 

1 

0 

-1 

-2 

0 

SI  12 

2 

1 

0 

-1 

-2 

0 

SI  13 

-2 

-1 

0 

1 

2 

0 

SI  14 

2 

1 

0 

-1 

-2 

0 

SI  15 

2 

1 

0 

-1 

-2 

0 

SI  16 

2 

1 

0 

-1 

-2 

0 

SI  17 

2 

1 

0 

-1 

-2 

0 

SI  18 

2 

-2 

151 


Question  Number 

A 

B 

C 

D 

E 

G 

PMC1 

2 

-2 

PMC2 

1 

1 

1 

1 

1 

1  0 

PMC3 

2 

-2 

PMC4 

2 

2 

2 

0 

PMC5 

2 

2 

2 

2 

2 

0 

Question  Number 

SA 

A 

N 

D 

SD 

NA 

PMC6 

2 

1 

0 

-1 

-2 

0 

PMC7 

-2 

1 

0 

-1 

-2 

0 

PMC8 

2 

1 

0 

-1 

-2 

0 

PMC9 

2 

1 

0 

-1 

-2 

0 

PMC  10 

2 

1 

0 

-1 

-2 

0 

PMC11 

2 

1 

0 

-1 

-2 

0 

PMC  12 

2 

1 

0 

-1 

-2 

0 

PMC  13 

2 

1 

0 

-1 

-2 

0 

PMC  14 

2 

1 

0 

-1 

-2 

0 

PMC  15 

2 

1 

0 

-1 

-2 

0 

PMC  16 

2 

1 

0 

-1 

-2 

0 

PMC  17 

2 

1 

0 

-1 

-2 

0 

PMC  18 

2 

1 

0 

-1 

-2 

0 

PMC  19 

2 

1 

0 

-1 

-2 

0 

152 


Question  Number  A  B  C  D  E 


GHI  J  KLMNOPQR 


PPE1 

2 

-2 

2 

-4 

PPE2 

-2 

2 

PPE3 

-2 

2 

PPE4 

2 

-2 

PPE5 

1 

1 

1 

1 

1 

1 

0 

PPE6 

1 

1 

1 

1 

1 

1 

111111  111110 

PPE7 

2 

2 

-4 

PPE8 

1 

1 

1 

1 

1 

1 

0 

PPE9 

-4 

0 

2 

4 

0 

PPE10 

1 

1 

1 

1 

1 

0 

Question  Number 

SA 

A 

N 

D 

SD 

NA 

PPE11 

2 

1 

0 

-1 

-2 

0 

PPE12 

2 

1 

0 

-1 

-2 

0 

PPE13 

2 

1 

0 

-1 

-2 

0 

PPE14 

2 

1 

0 

-1 

-2 

0 

PPE15 

2 

1 

0 

-1 

-2 

0 

PPE16 

2 

1 

0 

-1 

-2 

0 

PPE17 

2 

1 

0 

-1 

-2 

0 

PPE18 

2 

1 

0 

-1 

-2 

0 

PPE19 

2 

1 

0 

-1 

-2 

0 

PPE20 

2 

1 

0 

-1 

-2 

0 

PPE21 

2 

1 

0 

-1 

-2 

0 

PPE22 

2 

1 

0 

-1 

-2 

0 

PPE23 

2 

1 

0 

-1 

-2 

0 

PPE24 

2 

1 

0 

-1 

-2 

0 

PPE25 

2 

1 

0 

-1 

-2 

0 

PPE26 

2 

1 

0 

-1 

-2 

0 

PPE27 

2 

1 

0 

-1 

-2 

0 

PPE28 

2 

1 

0 

-1 

-2 

0 

PPE29 

2 

1 

0 

-1 

-2 

0 

PPE30 

2 

1 

0 

-1 

-2 

0 

PPE31 

2 

1 

0 

-1 

-2 

0 

PPE32 

2 

1 

0 

-1 

-2 

0 

PPE33 

2 

1 

0 

-1 

-2 

0 

PPE34 

2 

1 

0 

-1 

-2 

0 

PPE35 

2 

1 

0 

-1 

-2 

0 

153 


Question  Number 

A 

B 

C 

D 

E 

B 

G  H  i 

SMI 

2 

-2 

0 

SM2 

-2 

-4 

2 

-2 

SM3 

0 

2 

-2 

SM4 

Not 

Included 

in 

the 

Model 

SM5 

Not 

Included 

in 

the 

Model 

SM6 

Not 

Included 

in 

the 

Model 

SM7 

-2 

2 

4 

-4 

SM8 

1 

1 

1 

1 

1 

1  1  1 

0 

SM9 

2 

2 

2 

SM10 

1 

1 

2 

1 

1 

1  0 

Question  Number 

SA 

A 

N 

[ 

) 

SD 

NA 

SM11 

2 

1 

0 

-1 

-2 

0 

SM12 

-2 

-1 

0 

1 

2 

0 

SM13 

2 

1 

0 

-1 

-2 

0 

SM14 

2 

1 

0 

-1 

-2 

0 

SM15 

2 

1 

0 

-1 

-2 

0 

SM16 

2 

1 

0 

-1 

-2 

0 
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Question  Number 

A 

B 

|c 

D 

E 

G 

RC1 

Not 

Included 

in 

the 

Model 

RC2 

Not 

Included 

in 

the 

Model 

RC3 

Not 

Included 

in 

the 

Model 

RC4 

Not 

Included 

in 

the 

Model 

RC5 

2 

1 

-1 

-2 

RC6 

2 

-2 

-1 

RC7 

-1 

1 

0 

RC8 
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0 
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APPENDIX  D.  SOFTWARE  PROJECT  MANAGEMENT 
EVALUATION  MODEL  IN  DETAIL 


TABLE  OF  QUESTIONS  AND  SCORES  USED  TO  ESTABLISH  SCALING 


AND  SHIFTING  FACTORS 


Sub  Project 

Management  Area 

Number  of 
Questions 

Lowest 

Score 

Highest 

Score 

Difference 

between 

the 

highest 
and  lowest 

score 

Communication 

23 

-38 

66 

104 

Teamwork 

30 

-54 

73 

127 

Leadership 

17 

-34 

34 

68 

Organizational 

Commitment 

26 

-50 

62 

112 

Project  Manager 

27 

-52 

60 

112 

Stakeholder 

Involvement  -  Contract 

16 

-30 

42 

72 

Stakeholder 

Involvement  -  Market 

12 

-22 

34 

56 

Staffing  and  Hiring 

29 

-52 

64 

116 

Requirements 

Management 

27 

-50 

73 

123 

Project  Monitoring  and 
Control 

19 

-32 

54 

86 

Project  Planning  and 
Estimation 

35 

-70 

104 

174 

Scope  Management 

16 

-26 

45 

71 

Configuration 

Management 

13 

-22 

38 

60 

Quality  Engineering 

20 

-36 

57 

93 

Risk  Assessment  -  No 
Subcontracting 

19 

-34 

57 

91 

Risk  Assessment  -  With 
Subcontracting 

20 

-38 

63 

101 

Risk  Control 

17 

-26 

28 

54 
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SHIFTING  FACTORS  AND  SCALING  FACTORS 


Sub  Project  Management  Area 

Shifting  Factor 

Scaling  Factor 

Communication 

38 

10/104 

Teamwork 

54 

10/127 

Leadership 

34 

10/68 

Organizational  Commitment 

50 

10/112 

Project  Manager 

52 

10/112 

Stakeholder  Involvement  -  Contract 

30 

10/72 

Stakeholder  Involvement  -  Market 

22 

10/56 

Staffing  and  Hiring 

52 

10/116 

Requirements  Management 

50 

10/123 

Project  Monitoring  and  Control 

32 

10/86 

Project  Planning  and  Estimation 

70 

10/174 

Scope  Management 

26 

10/71 

Configuration  Management 

22 

10/60 

Quality  Engineering 

36 

10/93 

Risk  Assessment  -  No 

Subcontracting 

34 

10/91 

Risk  Assessment  -  With 

Subcontracting 

38 

10/101 

Risk  Control 

26 

10/54 
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SUB  PROJECT  MANAGEMENT  AREA  EQUATIONS 
Communication  Area  Evaluation  Model 

10  /  y3 

104  *  \  1 

\f»l 

Teamwork  Area  Evaluation  Model 

■30 


Communication  Area  Score  i 


C(  4  36 


T «tm werfe  Area  Score 


io  /  Vs 

" 127  *  Ik 


T(  4  S4 


Leadership  Area  Evaluation  Model 

In  the  leadership  section  of  SPMEI,  the  respondent  has  to  choose  to  respond  to 
one  of  two  questions:  LI 7  and  LI  8.  If  the  project  team  mostly  consists  of 
inexperienced  staff  then  the  respondent  should  answer  question  LI 7.  If  the 
project  team  mostly  consists  of  experienced  staff,  then  the  respondent  should 
answer  question  LI 8.  The  choices  for  these  questions  are  identical.  However, 
the  scoring  is  different.  The  model  for  both  cases  is  presented  below.  If  the  team 
mostly  consists  of  inexperienced  staff,  then  the  leadership  area  model  is  as 
follows: 


Lead  era  hip  Area  Score ' 


. .  /«■!? 


4  34 


If  the  team  mostly  consisted  of  experienced  staff,  then  the  leadership  area  model 
is  as  follows: 


Leadership  Area  Score  ■ 


.  „  /n«i& 

i”(.S  - 


4  iu  4  34 


Organizational  Commitment  Area  Evaluation  Model 

■2& 


Org  anizati  oml  Commitment  Area  Score  * 


10  /  v 


0Ct  4  £0 


Project  Manager  Area  Evaluation  Model 

Project  Manager  Area  Score  ■  PMt  4  S2 


10  f 

112  ^5' 


Stakeholder  Involvement  Area  Evaluation  Model 

In  the  stakeholder  involvement  section  of  SPMEI,  the  questions  after  S1 10  are 
divided  into  two  sections.  If  the  project  is  developed  for  the  market  without  a 
specific  contract,  then  the  respondent  should  answer  questions  Sill  and  S1 12.  If 
the  project  is  developed  under  a  contract  with  a  customer,  then  the  respondent 
should  not  answer  the  questions  Sill  and  S1 1 2,  but  the  questions  from  S1 1 3  to 
S1 18  instead.  If  the  project  is  developed  for  the  market,  then  the  stakeholder 
involvement  area  model  is  as  follows: 


Stakeholder  involvem  entArea  Score 


„  „  /»■  12  ■> 


If  the  project  is  developed  for  the  market,  then  the  stakeholder  involvement  area 
model  is  as  follows: 
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Stakeholder  Involvement  Area  Scores  — x  (  \  Slt  +  \  Slt  +  30  j 


Staffing  and  Hiring  Area  Evaluation  Model 

/»«2? 
10  It: 

Staffing  ana  Hiring  Area  Score ' 


Requirements  Management  Area  Evaluation  Model 


Requlrments  ManagmentArea  Score 


Project  Monitoring  and  Control  Area  Evaluation  Model 


Prefect  Monitoring  and  Control  Area  Score 


10  /  v 

MS 


PMCt  4  32 


Project  Planning  and  Estimation  Area  Evaluation  Model 

£*C? 


Prefect  Planning  and  Estimation  Area  Score  i 


174  V  L* 

M"1 


PPEt  +  70 


Scope  Management  Area  Evaluation  Model 


Scope  Hanag  em  entArea  Score  i 


Configuration  Management  Area  Evaluation  Model 


Configuration  Management  Area  Score 


10  /  v 

MS 


CMt  +  22 


Quality  Engineering  Area  Evaluation  Model 

-i2d 


Qualtt)*  Engineering  Area  Score  ■ 


s-  z 

\  fBl 


<?ff{  +  36 


Risk  Assessment  Area  Evaluation  Model 

In  the  risk  assessment  section  of  the  SPMEI,  there  is  an  additional  question  at 
the  end  of  the  section  for  the  projects  in  which  subcontracting  is  used.  The 
question  identifier  is  RA20.  If  the  project  does  not  utilize  subcontracting,  then  the 
risk  assessment  area  model  is  as  follows: 


Risk  As  sessment  Area  Score ' 


•h  34 


If  the  project  utilizes  subcontracting,  then  the  risk  assessment  area  model  is  as 
follows: 

■Stt  \ 


Risk  Assessment  Am  Score  ■ 


10  / 

100  *(  2- 


'■  W 


tfAf  4  3§ 


Risk  Control  Area  Evaluation  Model 

In  the  risk  control  section  of  the  SPMEI,  there  are  four  questions  that  are 
excluded  from  the  evaluation  model:  RC1,  RC2,  RC3,  and  RC4.  These  questions 
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are  included  in  the  instrument  to  enable  a  consistency  check  among  the 
responses  and  for  other  research  purposes.  Therefore,  for  the  risk  control  area 
model,  only  the  responses  from  RC5  to  RC17  are  included  in  the  evaluation 
model: 


Risk  Control  Area  Score  - 
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APPENDIX  E.  METRIC  FEEDBACK  INSTRUMENT 


Consent  to  Participate  in  Research 

Introduction.  You  recently  participated  in  a  research  study  entitled  “The 
Effectiveness  of  Software  Project  Management  Practices”  conducted  by  the  Naval 
Postgraduate  School.  You  are  now  invited  to  provide  feedback  on  the  study. 

Procedures.  The  data  you  provided  has  helped  assist  with  the  development  of  a 
software  project  management  effectiveness  (PME)  metric.  The  research  into  the 
usefulness  and  the  applicability  of  this  metric  is  ongoing.  The  PME  metric  for  your 
project  was  calculated  from  the  responses  in  your  survey  and  provided  to  you.  The 
purpose  of  this  additional  survey  is  to  inquire  into  the  usefulness  and  applicability 
of  the  original  survey  and  the  PME  metric  so  that  it  can  be  improved  in  the  future. 

Voluntary  Nature  of  the  Study.  Your  participation  in  this  study  is  strictly 
voluntary.  If  you  choose  to  participate  you  can  change  your  mind  at  any  time  and 
withdraw  from  the  study.  You  will  not  be  penalized  in  any  way  or  lose  any  benefits 
to  which  you  would  otherwise  be  entitled  if  you  choose  not  to  participate  in  this 
study  or  to  withdraw. 

Potential  Risks  and  Discomforts.  The  potential  risks  of  participating  in  this 
study  are: 

b)  A  breach  of  confidentiality  may  result  in  embarrassment  of  the  research 
subject. 

Anticipated  Benefits.  Anticipated  benefits  from  this  study  are: 

a)  To  assist  in  the  development  of  project  management  metrics  and  improve  the 

software  engineering  body  of  knowledge  to  improve  software  project 
management;  and 

b)  To  enable  the  development  of  a  tool  for  you  to  monitor,  evaluate  and  improve 

your  projects. 

Compensation  for  Participation.  No  tangible  compensation  will  be  given.  A 
copy  of  the  research  results  will  be  available  at  the  conclusion  of  the  experiment. 

Confidentiality  &  Privacy  Act.  Any  information  that  is  obtained  during  this  study 
will  be  kept  confidential  to  the  fullest  extent  permitted  by  law.  All  efforts,  within 
reason,  will  be  made  to  keep  your  personal  information  in  your  research  record 
confidential  but  total  confidentiality  cannot  be  guaranteed.  No  information  will  be 
publicly  accessible  which  could  identify  you  as  a  participant.  Research  records  will 
be  stored  and  maintained  in  electronic  form  on  NPS  secure  servers  only 
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accessible  by  the  researchers.  Any  hard  copy  material  containing  research 
findings,  including  a  thesis,  will  not  contain  any  personal  information. 

Points  of  Contact.  If  you  have  any  questions  or  comments  about  the  research,  or 
you  experience  an  injury  or  have  questions  about  any  discomforts  that  you 
experience  while  taking  part  in  this  study,  please  contact  the  Principal  Investigator, 
Dr  John  Osmundson,  831-656-3775,  josmundson@nps.edu.  Questions  about  your 
rights  as  a  research  subject,  or  any  other  concerns,  may  be  addressed  to  the 
Naval  Postgraduate  School  IRB  Chair,  CAPT  John  Schmidt,  USN,  831-656-3864, 
jkschmid@nps.edu. 

Statement  of  Consent.  I  have  read  the  information  provided  above.  I  have  been 
given  the  opportunity  to  ask  questions  and  all  the  questions  have  been  answered 
to  my  satisfaction.  I  have  been  provided  a  copy  of  this  form  for  my  records  and  I 
agree  to  participate  in  this  study.  I  understand  that  by  agreeing  to  participate  in 
this  research  and  signing  this  form,  I  do  not  waive  any  of  my  legal  rights. 
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Metric  Feedback  Instrument 

Please  read  the  Software  Project  Management  Effectiveness  Metric  provided  to 
you.  Then  hypothetically  consider  using  the  previous  survey  questions  on  the 
next  software  project  you  work  on  and  respond  to  the  following  questions. 

Manageability:  A  metric’s  information  should  be  available  and  concise.  How 
manageable  is  the  PME  metric?  (1  being  unmanageable  and  10  being  easily 
manageable) 

1  2  34  56  78  9  10 

□  □□□□□□□□□ 

Please  provide  any  comments  you  may  have  on  the  metric’s 
manageability: 


Meaningful:  A  metric  must  be  understandable  and  relevant  to  the  recipient  and 
provide  a  basis  for  decisions.  How  meaningful  is  the  PME  metric?  (1  being  not 
meaningful  and  10  being  very  meaningful) 

1  2  34  56  78  9  10 

□  □□□□□□□□□ 

Please  provide  any  comments  you  may  have  on  how  meaningful  the 
metric  is: 


Actionable:  Useful  metrics  information  makes  it  clear  what  response  is  needed, 
as  a  compass  makes  it  clear  whether  to  turn  left  or  right  or  stay  on  course.  How 
actionable  is  the  PME  metric?  (1  being  not  actionable  at  all  and  10  being  easily 
actionable) 

1  2  34  56  78  9  10 

□  □□□□□□□□□ 

Please  provide  any  comments  you  may  have  on  the  metric’s  actionability: 


Ambiguity:  Information  from  metrics  can  have  a  number  of  meanings  and  may 
be  misleading,  of  little  use,  or  downright  dangerous.  How  ambiguous  is  the  PME 
metric?  (1  being  very  ambiguous  and  10  being  completely  unambiguous) 

1  2  34  56  78  9  10 

□  □□□□□□□□□ 
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Please  provide  any  comments  you  may  have  on  the  metric’s  ambiguity: 


Reliability:  The  ability  to  trust  the  “instrument”  is  conditioned  on  the  reliability  of 
the  measurement.  How  reliable  is  the  PME  metric?  (1  being  completely 
unreliable  and  10  being  completely  reliable) 

1  2  34  56  78  9  10 

□  □□□□□□□□□ 

Please  provide  any  comments  you  may  have  on  the  metric’s  reliability: 


Accuracy:  A  reasonable  and  known  degree  of  a  metric’s  accuracy  is  essential. 
The  compass  showing  north  when  we  are  going  south  can  be  fatal.  How 
accurate  is  the  PME  metric?  (1  being  completely  inaccurate  and  10  being 
completely  accurate) 

1  2  34  56  78  9  10 

□  □□□□□□□□□ 

Please  provide  any  comments  you  may  have  on  the  metric’s  accuracy: 


Timely:  Measures  that  warn  of  a  disaster  after  it  has  happened  are  not  useful. 
Consider  measuring  the  PME  of  a  project  after  the  initial  requirements 
development  phase  of  a  project.  How  timely  is  the  PME  metric?  (1  being  far  too 
late  and  10  being  right  on  time) 

1  2  34  56  78  9  10 

□  □□□□□□□□□ 

Please  provide  any  comments  you  may  have  on  the  metric’s  timeliness: 


Predictive:  Some  metrics  information  will  signal  impending  problems  much  as  a 
drop  in  oil  pressure  is  the  harbinger  of  engine  failure.  Consider  measuring  the 
PME  of  a  project  after  the  initial  requirements  development  phase  of  a  project. 
How  predictive  is  the  PME  metric?  (1  being  completely  non  predictive  and  10 
being  very  predictive) 

1  2  34  56  78  9  10 

□  □□□□□□□□□ 
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Please  provide  any  comments  you  may  have  on  the  metric’s  predictability: 


Would  you  use,  or  would  you  like  to  see,  the  PME  metric  used  on  the  next 
software  project  you  work  on? 

Yes  No 

□  □ 

Please  briefly  explain  why  or  why  not. 


Thank  you  for  your  participation! 
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APPENDIX  F.  SOFTWARE  PROJECT  MANAGEMENT 
EFFECTIVENESS  METRIC  REPORT  CARD 


In  a  recent  study  on  the  effectiveness  of  software  project  management  practices, 
you  completed  a  survey  on  the  project  management  practices  in  place  on  a 
project  you  have  worked  on.  The  data  you  provided  has  helped  assist  with  the 
development  of  a  software  project  management  effectiveness  (PME)  metric.  The 
research  into  the  usefulness  and  the  applicability  of  this  metric  is  ongoing.  The 
PME  metric  for  your  project  was  calculated  from  the  responses  in  your  survey. 
For  your  information,  the  results  are  presented  here. 


Legend 

Area: 

Score: 

Average: 

Explanation: 


The  specific  area  of  software  project  management  measured. 
Calculated  rating  from  0  to  10  (0  being  the  lowest  and  10  being  the 
highest). 

Average  score  of  projects  in  the  study. 

Standard  comments  to  explain  the  area  score. 


Area 

Score 

Average 

Explanation 

Communication 

4.9 

6.9 

A  successful  project  requires  constant  and  effective 
communication  between  project  stakeholders.  This 
score  is  an  indication  of  the  effectiveness  of  the 

communication  in  the  project. 

Teamwork  6  1  7  0  As  so^ware 's  developed  by  teams,  strong  teamwork  is 

essential  to  successfully  completing  a  software  project. 
This  score  is  an  indication  of  the  effectiveness  of  the 
_ teamwork  in  the  project. _ 


Leadership  4.3  7.1 


In  a  software  development  environment  leadership  is 
how  personnel  in  management  positions  exert  social 
influence  to  enlist  the  aid  and  support  of  others  in  the 
accomplishment  of  project’s  goals.  This  score  is  an 
indication  of  the  effectiveness  of  the  leadership  in  the 
project. _ 


Organizational  7.1  7.1 

commitment 


Organizational  commitment  is  the  employee’s 
psychological  attachment  to  the  organization  and 
organizational  goals.  This  score  is  an  indication  of  the 
effectiveness  of  the  organizational  commitment  in  the 
project. _ 


Project  6.3  7.6 

Manager 


The  project  manager  position  is  a  key  role  in  a  software 
project’s  organizational  structure.  This  score  is  an 
indication  of  the  effectiveness  of  the  project  manager  in 
the  project. _ 


Stakeholder  4  4  6  6  Stakeholder  engagement  is  concerned  with  the  level  of 

involvement  of  all  the  different  stakeholders  during  the 
involvement  project  development  effort.  This  score  is  an  indication  of 

the  effectiveness  of  the  stakeholder  engagement  in  the 
_ project. _ 


Staffing 

and 

6.1 

6.5 

Staffing  and  hiring  is  the  ability  to  source  human 
resources  and  put  them  in  the  right  project  role.  This 

Hiring 

score  is  an  indication  of  the  effectiveness  of  the  staffing 
and  hiring  in  the  project. 
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People  5.6  7.0 


The  people  score  is  the  average  of  the  communication, 
teamwork,  leadership,  organizational  commitment, 
project  manager,  stakeholder  involvement,  staffing  and 
hiring  scores. _ 


Area 

Score 

Average 

Explanation 

Requirements 

management 

7.5 

6.7 

This  process  involves  the  management  of  the  software 
requirements  and  is  not  to  be  confused  with  the 
requirements  development  process.  This  score  is  an 
indication  of  the  effectiveness  of  the  requirements 
management  in  the  project. 

Project  6.5  6.6 

Monitoring  and 

Control 


Project  monitoring  is  the  process  of  keeping  the  project, 
project  related  factors  and  project  metrics  under 
continuous  observation.  This  score  is  an  indication  of  the 
effectiveness  of  the  project  monitoring  and  control  in  the 
project. _ 


Project  6.7  6.6 

planning  and 

estimation 


Project  planning  and  estimation  is  the  process  of 
establishing  and  maintaining  the  plans  that  define  the 
project’s  work  activities.  This  score  is  an  indication  of  the 
effectiveness  of  the  project  planning  and  estimation  in 
the  project. _ 


Scope  6.5  5.9 

management 


This  is  the  process  of  defining  the  scope  of  the  project 
and  keeping  track  of  any  changes  to  the  scope.  This 
score  is  an  indication  of  the  effectiveness  of  the  scope 
management  in  the  project. _ 


Process  6.8  6.4 


This  score  is  the  average  of  the  requirements 
management,  project  monitoring  and  control,  project 
planning  and  estimation  and  scope  management  scores. 


Area 

Score 

Average 

Explanation 

Configuration 

management 

10 

6.3 

Software  configuration  management  is  the  discipline  that 
enables  us  to  keep  evolving  software  products  under 
control,  and  thus  contributes  to  satisfying  quality 
constraints.  This  score  is  an  indication  of  the 
effectiveness  of  the  configuration  management  in  the 
project. 

Quality 

9.5 

7.1 

Quality  engineering  involves  ali  activities  put  in  place  to 
ensure  the  development  of  a  high  quality  product.  This 

engineering 

score  is  an  indication  of  the  effectiveness  of  the  quality 
engineering  in  the  project. 

Product  9  7  6  7  The  Pr°duct  score  is  the  average  of  the  configuration 

_ ' _ | _ management  and  quality  engineering  scores. _ 


Area 

Score 

Average 

Explanation 

Risk 

assessment 

6.0 

6.0 

Risk  assessment  involves  risk  identification,  risk  analysis 
and  risk  prioritization.  This  score  is  an  indication  of  the 
effectiveness  of  the  risk  assessment  in  the  project. 

Risk  Control  6  1  5  6  Risk  contr°l  involves  risk  management  planning,  risk 

resolution,  and  risk  monitoring.  This  score  is  an 
indication  of  the  effectiveness  of  the  risk  control  in  the 
_ project. _ 


Risk 


6.1 


Risk  management  is  an  inherent  aspect  of  any  software 
project.  This  score  is  the  average  of  the  risk  assessment 
and  control  scores. 
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PME  Score 


6.9 


6.5 


The  project  management  effectiveness  score  is  the 
weighted  sum  of  the  people,  process,  product  and 
risk  scores.  It  gives  an  indication  of  the 
effectiveness  of  the  software  project  management  in 
the  project. 
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APPENDIX  G.  FEEDBACK  METRIC  INSTRUMENT  RESPONSES 


Score 

Manageability  Comments 

9 

Nil 

9 

Nil 

7 

Nil 

4 

Nil 

8 

Nil 

10 

It's  a  short  list  easily  manageable  and  areas  and  overall  scores  can  be 
clearly  be  identified. 

4 

While  getting  back  the  "score"  was  interesting,  not  having  any 
explanation  of  why  or  how  the  score  was  derived  limits 

8 

1  am  not  sure  what  you  mean  by  manageable 

Score 

Meaningfulness  Comments 

7 

As  a  high-level  indicator,  1  think  it  clearly  defines  the  areas  of  good 
performance  and  the  areas  of  concern.  I'm  not  quite  sure  whether  we 
can  treat  these  metrics  as  values,  and  while  1  notice  that  the  final  score 
was  a  weighted  sum,  it’s  not  quite  apparent  which  areas  are  given  the 
most  weighting,  especially  since  the  unweighted  average  came  out  so 
close  to  the  final  score.  Realistically,  1  am  not  going  to  be  able  to  address 
each  of  the  low  scoring  areas  simultaneously,  so  if  1  have  to  pick  an  area 
of  improvement  1  want  to  pick  the  one  that’s  going  to  give  me  the  best 
chance  of  improving  my  project  success,  and  that  may  not  be  the  one 
with  the  lowest  score.  While  knowing  the  average  score  is  probably  a 
great  metric  for  you,  I'm  not  sure  what  1  can  do  with  it.  1  would  like  to  see 
target  bands  or  something  similar  that  that  gives  me  a  clearly  defined 
goal  to  aim  for.  I'm  not  too  concerned  about  beating  the  average 
(commercial  world  might  look  at  this  differently,  i.e.,  better  than  average 
makes  them  look  good  when  to  win  a  contract),  1  want  to  know  how  a 
certain  score  will  impact  my  project.  Perhaps  you  can  use  the  data  from 
your  surveys  to  work  out  some  bands,  it’s  entirely  subjective,  but  in 
reality  so  was  the  survey. 

7 

Nil 

8 

Without  the  average,  these  metrics  are  meaningless,  so  the  point  is  how 
significant  and  precise  is  the  average  ? 

7 

Nil 

8 

Nil 
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8 

The  scores  are  clear  and  simple;  you  can  easily  assess  how  well  the 
surveyed  group  performed  and  how  well  you  did.  The  areas  are 
straightforward  along  with  their  explanations.  However,  in  order  to  begin 
self  improvement  it  would  be  good  to  see  a  breakdown  of  key  techniques 
in  each  area  and  how  you  scored  on  each  if  it's  available.  That  you  could 
begin  focusing  on  some  techniques  that  you  were  lacking  in.  Though  this 
extra  information  may  work  adversely  against  the  manageability  of  the 
report  which  is  really  concise  at  the  moment. 

4 

For  some  of  my  scores,  1  know  enough  that  1  agree  -  good  score  for  CM, 
not  so  good  for  risk  management.  But  I'm  not  sure  why  1  got  a  poor  score 
for  Leadership. 

10 

Nil 

Score 

Actionability  Comments 

7 

As  before,  the  areas  where  we  need  to  improve  are  clear,  but  its  hard  to 
prioritize  which  ones  to  go  after  first. 

7 

When  a  project  is  performing  very  poorly  by  a  rather  large  gap,  the 
metric  provides  clear  insight;  in  other  cases  it  will  be  less  clear  what 
action  to  take. 

8 

Nil 

6 

Nil 

8 

Nil 

7 

1  touched  on  this  a  bit  in  the  last  question,  1  think  for  some  areas  that  are 
a  bit  broader  -  what  you're  doing  well  and  what  you  need  to  improve  on 
may  not  be  so  clear.  But  for  the  areas  that  are  based  on  a  technique,  like 
'Configuration  Management',  which  is  one  1  need  to  improve  on,  1  can 
clearly  see  how  to  improve  that.  And  coincidently  my  company  it 
currently  working  hard  to  improve  our  standard  of  configuration 
management. 

3 

Nil 

8 

Nil 

Score  Ambiguity 


7  I  don’t  think  the  PME  metric  is  ambiguous  but  I  think  it  is  still  lacking  in 
definition.  It  tells  me  how  I  scored  compared  to  the  average,  but  it  does 
not  tell  me  if  the  scores  are  good  or  not.  My  projects  final  PME  score 
was  higher  than  average,  but  I  know  we  performed  atrociously,  so  am  I 
to  assume  that  because  we  beat  the  average  we  are  actually  doing 
alright?  What  projects  did  you  use  to  create  your  average,  did  you  get  a 
good  spread  of  projects  or  just  a  whole  bunch  of  bad  projects?  Did  we 
only  beat  the  average  because  most  of  the  projects  sampled  were  bad 
projects? 
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6 

The  generated  report  is  unambiguous,  but  some  of  the  questions  in  the 
original  survey  could  have  a  wide  variance  in  interpretation. 

8 

Examples  would  remove  ambiguities  in  certain  cases 

3 

Nil 

8 

Nil 

8 

The  areas  create  clear  boundaries  and  the  scores  that  relate  to  them  are 
simple  to  understand.  1  think  there's  only  a  little  and  maybe  not  even 
crucial  ambiguity,  like  1  stated  in  the  earlier  questions,  where  within  a 
broad  area  such  as  'Communication'  you  may  not  know  what  you  are 
doing  right  and  what  you  need  to  improve  on  within  that  area. 

8 

Nil 

8 

Nil 

Score 

Reliability  Comments 

9 

1  am  convinced  that  the  scores  from  the  survey  will  produce  reliable 
metrics.  The  problem  comes  from  getting  a  reliable  source  of  data  to 
produce  the  metrics 

7 

Nil 

8 

Nil 

5 

Nil 

8 

Nil 

9 

The  survey  seemed  to  translate  well  into  scores  1  could  relate  to. 

5 

Nil 

8 

Nil 

Score 

Accuracy  Comments 

6 

here  is  an  inherent  inaccuracy  that  comes  from  doing  something 
subjective  like  a  survey  and  that  comes  from  the  bias  of  the  person 
completing  the  survey.  This  could  probably  be  combated  by  surveying 
a  wide  sample  of  people  from  the  same  project  and  getting  someone 
independent  to  make  an  assessment 

6 

Nil 

7 

For  some  cases  the  scale  of  assessment  could  have  been  a 
percentage  rather  than  a  five  level  scale 

7 

Nil 

8 

Nil 

9 

1  found  it  very  accurate.  The  areas  1  consider  myself  good  at  or 
lacking  were  reflected  spot  on  in  the  report. 

4 

Nil 

8 

Nil 
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Score 

timeliness  Comments 

7 

1  disagree  with  the  statement  that  measures  warning  of  disaster  after  it 
has  happened  are  not  useful.  In  my  opinion  it  depends  entirely  on  how 
you  intend  to  use  the  measures.  Back  on  topic,  this  metric  would  be 
considered  timely  if  produced  just  after  the  initial  requirements 
development  phase  of  a  project,  but  1  would  not  be  using  these  metrics 
as  a  predictive  tool. 

8 

Nil 

9 

Nil 

5 

Nil 

8 

Nil 

9 

1  think  you'd  need  that  real  grasp  of  what  work  is  to  be  carried  out  and 
the  risks  involved  with  getting  it  to  the  requirements  first.  So  1  think  this  is 
a  good  time. 

4 

Would  you  know  enough  to  get  meaningful  answers  to  the  questions? 
Just  because  a  good  CM  infrastructure  was  in  place,  suppose 
developers  stopped  using  CM  and  checked  in  code  updates  less  and 
less  often? 

8 

Nil 

Score 

Predictive  Comments 

6 

While  this  metric  could  possibly  be  predictive,  1  would  not  use  this  metric 
as  a  predictive  tool.  1  would  be  using  these  metrics  at  the  end  of  a  project 
as  a  way  of  assessing  how  we  did  in  the  project  and  identifying  what 
areas  we  need  to  improve  for  our  next  project.  1  think  there  are  better 
predictive  measures  out  there,  PSM  has  made  a  successful  business  out 
of  determining  the  best  measures  to  use  at  different  stages  of  a  project. 
As  far  as  1  know,  organizations  make  money  out  of  running  multiple 
software  projects,  they  are  not  normally  interested  in  starting  up  a 
company  to  complete  one  project  before  shutting  down  again 

7 

For  some  areas,  low  scores  will  be  very  predictive  of  problems  (e.g., 
Stakeholder  involvement).  Others  1  would  give  less  weight  to  that  early 
on  in  the  project  (e.g.,  Teamwork). 

9 

Could  be  proactive  if  the  PM  is  confronted  to  higher  risks  in  the 
upcoming  projects 

4 

Nil 

8 

Nil 

8 

This  is  a  tough  one.  It  reinforces  a  lot  of  management  techniques  that 
need  to  be  considered  at  the  requirements  stage  of  development  and 
has  the  chance  to  improve  predicting  aspects  of  the  project. 
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3 

Nil 

5 

Nil 

Yes/No 

Please  briefly  explain  why  or  why  not 

1 

This  is  the  sort  of  metric  you  need  to  use  at  the  end  of  each  project.  It 
clearly  identifies  strengths  and  weakness,  and  from  that  you  can  select 
areas  that  need  improvement.  You  can  compare  previous  results  with 
later  results  to  see  if  the  processes  you  have  introduced  have  actually 
resulted  in  improvements  to  those  weaker  areas.  1  think  you  have 
created  a  great  tool  to  analyze  strengths  and  weaknesses  of  an 
project/organization  but  your  metrics  report  probably  needs  a  bit  of 
work  before  you  could  use  it  as  a  predictive  management  tool.  This  tool 
is  perfect  for  use  at  the  end  of  a  project  because  you  can  use  it  to 
identify  areas  that  need  to  improve  to  make  your  next  project  work 
better. 

0 

No,  not  as  is.  The  original  questions  need  to  be  less  open  to 
interpretation.  1  would  likely  begin  with  a  metric  reduced  in  scope  to 
questions  that  could  be  answered  objectively,  then  as  the  team  is 
allowed  time  to  build  trust  and  establish  relationships,  introduce  the 
more  subjective  question  areas 

1 

At  least  1  can  evaluate  whether  the  process  is  evolving  or  not.  and  also 
helps  to  fix  problems  per  domain 

1 

Nil 

1 

Nil 

1 

Yes.  1  found  the  survey  very  useful  as  a  chance  to  reflect  on  my  last 
project  and  I've  found  it's  made  me  think  about  my  current  projects. 

0 

Since  not  feedback  was  given  on  how  survey  answers  were  converted 
into  PME  metrics,  it  is  not  possible  for  me  to  understand  what 
behaviors  were  good  and  which  behaviors  need  to  be  changed.  Also, 
as  for  the  leadership  questions,  senior  leadership  is  not  something  that 

1  or  a  typical  project  leader  can  really  influence,  control,  or  change. 

1 

Yes,  if  the  metrics  is  available  throughout  the  projects 
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